How does scalability work

We run stateless services that can be easily scaled up or down whenever possible. Configuring how and when to autoscale ahead of time is much easier than having to manually rescale when traffic changes drastically. It also simplifies many other aspects of maintaining a service, so it is almost always worth setting services up this way from the beginning when possible.

For most other scaling optimizations, finding the balance between not over-engineering early and being ready in time to support more load is the trickiest part.

Some clear indicators that a service needs some love are multiple shards running hotter, slightly degraded performance at peak traffic times or degraded performance for individual power users and groups. Our specific strategies depend on the specific problem. We recently built a service that stands in front of our messages database that now coalesces requests and will allow us to easily add other optimizations in the future.

Another is a re-architecture of our guilds service, which as our biggest communities have grown, started struggling to handle , connected sessions per Elixir process. We horizontally scaled the Elixir processes so that we can scale as the number of connected sessions increases. Many of our stateful services that maintain connections to clients, and process and distribute messages in real-time, are written in Elixir, which is well suited to the real-time message passing workload.

But in recent years we have run up against some scaling limitations. These include ZenMonitor for coalescing down monitoring, Manifold to batch message passing across nodes, and Semaphore, which is helpful for throttling as services get close to their limits. ScyllaDB and Rust are two examples where our explorations have paid off and they have been good solutions to some of our problems.

To do that, they adopted a hybrid cloud infrastructure and run the website from a private and public cloud. Designing and building solutions that focus on key customer problems without introducing unnecessary complexity or less desired user experience is critical to delivering high-quality products and services that can enable and support business growth.

Solutions we build should be both performant and flexible to address business needs. Generally speaking, there are three options for technology scaling: horizontal duplication, splitting the system by function, and splitting the system into individual chunks.

This is why we adopted a hybrid cloud architecture: running the site from both the private and public clouds. Private cloud provides us with predictable performance and cost, and public cloud ensures a virtually unlimited capacity for scaling out as needed. Our engineers write and then own their code in production. This continued ownership increases quality and accountability.

The goal of the site operations teams is to enable developers to maintain this ownership. Automation helps to move the focus from performing repeatable tasks manually to automating these and focusing on more important strategic work. At Udemy, we adopted the infrastructure-as-code approach to managing infrastructure.

Because these tools are vendor agnostic, they work equally well in public and private cloud implementations. This allows us to use the same toolkit across multiple cloud platforms reducing our efforts to manage these platforms. Sandeep Ganapatiraju, a lead software development engineer, said the goal of scalability at the e-commerce platform BigCommerce is to increase customer usage with the least amount of technical changes. Multiple simulations of customer types and usage help them plan out when to release new business features.

This means having a clear strategy by simulating various customer usage scenarios upfront based on how the business sees usage likely to grow over the next few months, including product releases from alpha, beta, and becoming generally available. Scalability is more about the ability of the platform to adapt to increasing customer usage with the least amount of changes needed. How do you build this technology with scalability in mind? Share some specific strategies or engineering practices you follow.

We do multiple simulations. We create examples of the largest user creating even larger usage, an average user operating at average usage and also, a sudden short burst of usage spike such as a sale. Each of these scenarios is specifically thought about while keeping business in mind.

Then, we keep the business informed to help a planned rollout of new features. BigCommerce uses JMeter and queries to explain plans for quick checks. Then we use BlazeMeter once we have a solid test laid out to test overall system performance. We do a lot of monitoring of production using Grafana dashboards to see how various services are performing in production.

We use also do canary deployments to release a new version to a specific software subset of users to see any unprecedented bugs or scales. We also monitor requests that take a long time with distributed tracing tools to see which specific microservice is slowing the overall request.

At Sovrn , scalability is crucial since their network handles tens of thousands of API requests per second. Engineering Manager Theo Chu said they must anticipate future demand when building in order to prevent latency and keep customers happy. Scalability means building our technology and products from the ground up with future scale in mind. It means anticipating future demand and being able to meet that demand without having to re-engineer or overhaul our system. Scalability is especially crucial for us since we handle tens of thousands of API requests per second.

Our data pipelines process and analyze billions of events per day. We support traffic from major publishers on the web and our partners expect us to handle their scale on a daily basis without sacrificing latency or reliability.

Building for scalability means designing, building and maintaining engineering systems with a deep technical understanding of the technologies that we use and the performance constraints of our systems. Our approach has been to build for scalability from the bottom up through both technical and product perspectives. On the technical front, we rely on underlying technologies and frameworks that enable scale. On the product front, we find that building foundational components early on for anticipated scale trades off much better than having to re-architect the system later on.

This was exemplified with our Optimize product, where we have been able to scale effortlessly after designing our database to handle hundreds of millions of mappings in earlier iterations. We rely on a range of technologies that support and process data at scale, including Cassandra, Kafka and Spark. Since these are the foundational blocks of our system, we optimize them heavily and load test each component up to multiples of our current scale to enable scalability and address any bottlenecks.

Since our infrastructure is fully on AWS, we also utilize AWS tools that support scalability such as autoscaling groups. Optiver , a global electronic market maker, uses their own capital at their own risk to provide liquidity to financial markets.

Optiver's engineers and traders come together to craft simple solutions to complex problems. Will Wood, based in Chicago, said that to keep tech scalable, his team uses load testing systems they build themselves to see how their tech reacts in specific situations. Scalability means being able to easily handle the next busy market day.

The technology I'm building is at the center of the environment, so if it has performance problems, the impact will be large. If it scales well, the firm will be able to remain fully active through extreme market conditions. My goal is that this technology has the same performance characteristics on an extreme day as it has on an average one.

I try to reduce the variance in performance as much as possible. In order to accomplish this, I choose algorithms with consistent performance, use simple programming language features and keep the resources my systems use isolated from interference. I also design my systems to handle a specific load, which is usually some estimate of an extreme day, and behave in a deterministic way if that load is exceeded.

Then, I monitor the actual performance and load in the production environment, signaling me when the latter is approaching the designed threshold. On my team, we build our own systems for load testing.

This allows us to test very specific scenarios and adapt to changing market conditions and business requirements. We have also developed a system for monitoring the performance of our applications in the production environment. This gives us regular feedback into how our tech is behaving and allows us to quickly notice any degraded performance. Bugosh says his team relies on data from previous years, frequent load testing and tools like AWS to help ensure their systems are ready for the busy season.

At Jellyvision, we face a fairly interesting set of scalability problems. Our ALEX Benefits Counselor product is used by employees to help them choose their healthcare each year during open enrollment.

Since most of our customers have open enrollment during the same few months in the fall, we experience a spiked but predictable load pattern. The ability to have those systems scale up and down and deliver results quickly is key for our ability to help our users. The back-end services utilize autoscaling, in case we see any large unexpected spikes in traffic, but are efficient during our off-peak times. The first and most important strategy we have is identifying the expected usage of our systems when we are initially developing the requirements.

Those requirements guide our expectations during code and design reviews and help us to choose the right core technologies and architecture patterns for our systems.

This process leads us to make deliberate choices about what logic should live in the front-end JavaScript applications and what logic should live in our back-end services.

Since our back-end services operate mostly independently, we can add instances at will. The other part of our strategy depends on having years of data around our expected load levels, based on the number of customers and historic load levels. We perform extensive load testing leading up to our peak times and will adjust our baselines so that autoscaling has less work to do.

We use Cloudfront and S3 together to serve our front-end JavaScript code and media assets. For logging and metrics, Sumo Logic is our vendor. Fintech company HRT has built its own custom tools and proprietary reliability features to help with scalability issues. Scalability means understanding how current capacity is being utilized and how technology will behave as it approaches or exceeds saturation.

This is particularly important when operating in financial markets, where volume varies greatly and bursts of activity can unexpectedly overwhelm a system that appeared to have substantial headroom.

In finance, opportunities can be fleeting. Failure to scale to meet growth demands can quickly reflect on the bottom line.

In proprietary trading, there are several dimensions of growth to consider: market volume, geographies and asset classes, our catalog of trading models and sophistication of machine learning. Our success depends on being able to respond to expansion in all of these dimensions.

Building scalable technology starts with company culture. We strive to hire developers that can reason about complex systems and foster an engineering environment that values a long-term mindset. We cultivate that mindset through openly collaborative development and an iterative code review process where we encourage scalability and maintainability.

Furthermore, we leverage an expansive compute cluster for research and development. This organically encourages modular design by improving parallelism in the cluster and increasing productivity. Since we run the same software in production, those aspects of scalability carry over nicely to the live environment.

Meanwhile, a collection of automated stress tests provides continual insight into performance and scaling considerations. A foundational element of our technology that facilitates scale is a set of refined libraries that provide efficient communication among components.

And as a startup owner, you need to keep up with those needs to keep your business running. This is where software scalability comes in. In short, you need to build your product with growth in mind.

So why do you need to make software scalable? Scalable software can remain stable while adapting to changes, upgrades, overhauls, and resource reduction. To further understand this, here is an example. You can upgrade its security features and integrate other networks to increase its usability.

With improved quality, you can sell it as a premium subscription and offer it to larger enterprises. Scalability matters when it comes to selling software. Your competitors are always after the next big thing, and you have to keep up with them.

The more innovative you are with your product, the more people will patronize your brand. Software scalability allows your business to remain stable and productive despite the volatile nature of the industry. Furthermore, you raise profit by selling the premium version of your products. With a reliable product, you can easily hop on the latest trends and influx of demand. Likewise, your business also benefits in the long run, with lower maintenance costs, higher agility, and cheaper upgrade expenses.

This adds speed while reducing strain on the system. SQL does scale read operations well enough, but when it comes to write operations it conflicts with restrictions meant to enforce ACID principles. Platform-as-a-service relieves a lot of scalability issues since the PaaS provider manages scaling. Scaling can be as easy as upgrading the subscription level. Function-as-a-service evolved from PaaS and is very closely related. Serverless computing provides a way to only use the functions that are needed at any given moment, reducing unnecessary demands on the back-end infrastructure.

FaaS is still maturing, but it could be worth looking into as a way to cut operational costs while improving scalability. Prioritizing scalability prepares your business for success.

Set up a free appointment with one of our developers to talk about where you need to go and how we can get you there! Contact Us. Software Engineering. What is Software Scalability? The Benefits of Scalable Software Scalability has both long- and short-term benefits. That puts the company at risk for a data breach or worse. Scalability gets pushed aside in favor of speed, shorter development cycles, or lower cost. Principles of Scalability Several factors affect the overall scalability of software: Usage Usage measures the number of simultaneous users or connections possible.

Increasing it should be as simple as making more resources available to the software. Maximum stored data This is especially relevant for sites featuring a lot of unstructured data: user uploaded content, site reports, and some types of marketing data. Code Inexperienced developers tend to overlook code considerations when planning for scalability.

It also takes time to implement the new hardware. Scaling happens faster, too, since nothing has to be imported or rebuilt. Here are some best practices for incorporating scalability from the start: Use load balancing software Load balancing software is critical for systems with distributed infrastructure like horizontally scaled applications.

Examples would include how well a hardware system performs when the number of users is increased, how well a database withstands growing numbers of queries, or how well an operating system performs on different classes of hardware.

Enterprises that are growing rapidly should pay special attention to scalability when evaluating hardware and software. Download eBook. Learn More. Join your peers for the unveiling of the latest insights at Gartner conferences. Expert insights and strategies to address your priorities and solve your most pressing challenges.