Architectures for Modern Streaming Applications

#Overview

Building streaming applications are no longer reserved for industry heavyweights. Streaming applications are being leveraged by governments, companies, cultural institutions to better connect with their users and create a digital archive of video content. Modular, API-first tech stacks enable this to be possible without such a steep price tag as monolith solutions and enable greater flexibility.

Modern streaming applications using a modular tech stack enable teams to pull in existing content from external data sources. They can easily manage and process data streams - and then enrich metadata using tools such as a CMS to make quick changes simple and straightforward. This data can then be served across multiple frontends from a single source. Modular architectures allow teams to access all of the functionality that they need to create a high-quality streaming service and leverage the expertise of specialized services. Using these best-of-breed services enables a greater capacity for the in-house development team as well as more flexibility in the future.

To understand how this would work in practice, explore our Case Study with Telenor.

Beginning to plan the API-first, modular architecture for a streaming application can be a daunting task. That’s why we have created some high-level architectures to give you a place to start and serve as a guide for building an optimized modular tech stack for a streaming application.

#Architecture

#Architecture Breakdown

Authentication/Authorization

Authentication and Authorization software center around confirming the identity of users and ensuring that only the correct people have access to the system.

Global Cache

Global caching improves performance by storing copies of files or data in regions that are globally distributed. Future requests of the same data are pulled from the region closest to the request enabling the data to load more quickly. This can be particularly helpful for content that is accessed repeatedly. Cloudflare is a popular option due to its robust network of data centers.

CDN

A content delivery network (CDN) is a globally distributed group of servers that allow for quick transfer of assets. CDNs reduce hosting bandwidth and can improve security by pulling content closer to website visitors.

Search

Adding in search functionality to your final digital product can be a great way to improve user experience and ensure that your content is more visible. There are several popular solutions for Search including Yext or Elasticsearch.

Metadata

Metadata is data about a specific piece of data, such as an asset, that makes it easier to search for, track, and organize. Taking the example of an asset, metadata may include where and when the asset was uploaded, as well as keywords describing what the asset shows.

Marketing Tools

Marketing tools cover a wide range of tools that can be helpful to a team to give them the functionality they need and exactly the insights that they require. Marketing tools can range from personalization to feedback, to business insights tools. These tools are highly dependent on the needs of your team and resources but are helpful for gathering data and user interactions.

Data Management and Processing

Data management and processing for a streaming application can encompass the various aspects of message ingestion, storage data, data aggregation, transformation, and structure before it can be analyzed by any analytics tools. Depending on the architecture, this can be several different tools that accomplish this step or they can be handled via platforms such as Upsolver.

Stream Processor

A stream processor takes data from a source and translates it into a standard message format, and streams it on an ongoing basis. Kafka is a popular tool for stream processors and for setting up an event streaming platform.

Data Sources

A data source for a streaming application could be a wide range of things. It refers to the existing external systems where various data can be pulled in order to enrich the streaming application. Connecting existing data sources can enable teams to create data-rich applications without the need for laborious content migrations over and over again.

Asset Management

Asset Management tools make it easy to upload, transform, manipulate assets for your website or digital project. These services give users more control over their assets, a critical component of any modern digital project. Popular tools for asset management include Filestack and Fastly.

User Generated Content

User-Generated Content (UGC) is any content that is created by people rather than companies or branding companies. UGC could be anything from reviews to text to images created by users. In the context of a streaming app, this could include a variety of variables - such as preferences, watchlists, watch history, etc.

Frontend(s)

The frontend is where data is transformed into an accessible interface where users can view and interact with the information. This can come in a variety of forms and channels but the key point is that the user and their experience are at the core of frontend work.

#Architecture Best Practices

In order to get the most out of your modular, API-first tech stack for a modern web application, there are some critical best practices that should be considered.

Allocate sufficient time for an evaluation period

Modular architectures for streaming applications require careful construction so that the correct information flows freely to create an optimal user experience. This requires careful planning at the forefront to understand how the various systems will communicate with one another securely. Adding in a system such as a CMS that is specifically intended for users to edit metadata can be helpful in improving the developer experience and keeping data up-to-date.

Ensure the redundancies are intentional

While it may be the case that you want to build some redundancies into your tech stack for extra security, it is important to make sure that they serve a real purpose. Creating too many redundancies or unnecessary ones can lead to a bloated tech stack rather than an agile, flexible system.

Decentralize data sources

To build and enrich a streaming platform, it is very likely that content will be coming from several external sources. By breaking down the monolith approach and choosing specialized, best-of-breed services, teams ensure that they have all of the functionality they need without being weighed down by unnecessary dead weight. While it may take time to find the correct balance of services, the time investment to have a customized, well-functioning architecture is worth the investment. When systems are communicating via API, it becomes much easier to add and remove services without disrupting existing data.

Consider data privacy standards when choosing services

Data privacy varies wildly from region to region and it is important to consider how various services will handle data. For example, the EU has much stricter data privacy laws than other countries. Ensuring that data is stored in the regions that match your standards can be a key consideration when choosing your tech stack.

Explore the complete Architecture Series

• Architectures for Modern Websites
• Architectures for Modern Web Applications
• Architectures for Modern eCommerce Applications
• Architecture for a Video Streaming Service
• Architectures for Federated Content Hubs