BroadcastOps: DevOps in media

Jannik Altgen
Jannik Altgen
Business Consultant
Tags
Broadcaster
DevOps
Agile Development
Consulting
Change
Published 2. December 2022

How Broadcasters can establish ways to work with the change instead of against it using DevOps

Tech stacks at TV broadcasters used to rely on custom hardware, often monolithic in architecture, supplied by a few vendors locking down the market. But that has changed: Performant COTS hardware and convergence on IP for media transport have initiated the move to define many services in software rather than specialized hardware.

Using agile methods and DevOps principles, software release cycles have sped up dramatically. Consequently, with more and more services being run in software broadcasters are hard-pressed to level up their game, to keep up with the changing landscape and establish ways of working with the change instead of against it.

I. The need

Broadcasting is an industry driven by long lifecycles and almost no downtime. Many stations are public institutions run by long-term employees. The industry largely operates with monolithic, tightly coupled architectures, most of which are on-premises. In addition to operational security, data security needs are high – the personal data of whistleblowers that may be stored could turn out to be life-threatening. Designing systems in this environment is a challenge.

If technological change is a wave, broadcasters are trying to surf it, but carefully. It can be difficult to stay ahead of the curve when integrations into decades-old systems must be maintained in tightly regulated spaces. Broadcasters are hard-pressed to keep up with the increasing pace of software lifecycles imposed by manufacturers pivoting to DevOps methods. i.e., the collaborative or shared approach to the tasks performed by the company’s application development (dev) and IT operations (ops) teams. In its most narrow interpretation, DevOps describes the adoption of iterative software development, automation, and programmable infrastructure deployment and maintenance.

The increase in COTS hardware performance along a growing maturity of software implementations has made available a large class of "software defined" products, solving problems that previously required expensive custom hardware. Software defined vision mixing, audio mixing, video routing, graphics playout, channel playout – there is a piece of software that does it. Thus, broadcasters are not simply having to increase adoption speed for software that they already have – they also face the challenge of having to replace traditional hardware with "COTS plus software-defined" solutions.

Broadcasters need to adapt fast to this change in their environment or face the challenge of too-large change processes if opportunities are missed.

DevOps integrates teams that are traditionally separated: software development, quality assurance and IT operations.

II. Defining BroadcastOps

Let's define what we'll call BroadcastOps. The idea is to iterate through given methods of the Agile framework as well as DevOps best practices and consider how they might apply in a broadcast environment to help adapt to the increasing speed of the software landscape.

1. Continuous Integration and Development
Continuous Integration and Continuous Deployment (or CI and CD), the on-going task of largely automatic testing of design changes and their deployment to production environments, is something that is almost never done in top-down systems design at broadcasters. Strict Requirements Engineering is the norm, disregarding the shortcomings of this model, especially in organizations where stakeholders are as numerous and diverse as in media publishing houses.

Instead of overlooking or missing requirements, planning year-long Gantt charts and writing lots of change-requests, changes must be continuously integrated into the broadcast production chain. This move goes away from having to flick one big switch at the end, replacing and integrating components on a smaller level. Think "Microservices at the infrastructure level".

Clearly, to make this viable, systems must be automated to the point where version up-and-down steps can be run at least overnight, as opposed to time-consuming year-long projects for version changes.

Here are a few specific ideas to where this can apply at broadcasters:

  • Automated testing for network configurations, possibly using a digital twin such as GNS-3 or EVE-NG (network simulators)
  • Automated route-checker to test if video streams can reach their target, again digital-twining for video routing
  • Essentially, running a network simulation and testing changes against it after having defined which endpoint must talk to whom – like a digital twin

2. Infrastructure as Code

Systems are almost never defined in machine-readable manner. CAD drawings created manually, excel sheets maintained manually, maybe there are some config databases, but the data is rarely used operationally. This data must be made usable for orchestration.

  • Automatic documentation diagrams using declarative methods that could result from Infrastructure-as-Code (IaC) definitions (e.g., Mermaid.js diagrams)
  • Manage and use machine-readable data about endpoints to provision and configure cloud resources, when needed, based on IaC best-practice.
Get this white paper
Would you like to learn more?
Enter your data here and click on "Download now“. You will receive a download link to your email address afterwards. Please note that the delivery may take a few minutes.
Request instant free download access to the full white paper
BroadcastOps – DevOps for media verticals
Agree to data processing*