Skip to main content
Login Contact Us Contact Us

Blog

Blog

Recovering from Error States in Hybrid GV Orbit/AMPP Systems (Part 1)

Article written by Steve Bilow, Senior Product Manager, Infrastructure SW Platforms 

Average reading time: 7-9 minutes

Introduction

Recent enhancements to Grass Valley orchestration tools enable a variety of creative solutions to problems in the media industry. Among the most exciting of these are those that enable users to leverage the power of the logic engine underlying what is known as “MapView” to integrate the GV Orbit media orchestration system, the Agile Media Processing Platform (AMPP), and the significant probing and display capabilities of the Kaleido multiviewer and its AMPP-based equivalent to solve the problems of potential hardware failures and disaster recovery.  We will look at how to do these things but first we need a few definitions.

What is GV Orbit?

GV Orbit is a dynamically configurable media orchestration solution that provides a unified user experience for configuration, control, and monitoring in SDI, IP, and Hybrid facilities. It runs on virtual or physical servers.

What is MapView?

GV Orbit MapView is part of the toolkit for building GV Orbit control and monitoring projects. It allows users to create extremely flexible customizable UI screens; thus, providing a scalable control and monitoring user experience for Grass Valley and for third-party devices. MapView allows users to observe many channels with large amounts of status information. That data may be used in nearly infinite ways. One that is fundamental to the upcoming examples is monitoring by exception. MapView control, monitoring, and status screens are viewable on any client computer’s display as well as on Grass Valley multiviewer video walls. This, too, you will see below.

MapView High-Level Concepts Summarized
The devices in your system send log messages that may either be purely informative or may carry alarm information. In the latter case, you can highlight those messages on a graphical monitoring screen that you create with MapView.

Generally, a device sends log information to a logging service. From there, status messages are moved to other GV Orbit services which aggregate status and alarm messages and then publish them. A GV Orbit MapView client can subscribe to these and use them. With this data, a user can create graphical custom tiles to filter and monitor the status and alarms from these devices. Then the data can trigger custom actions based on the status parameters of any combination of devices. MapView includes an extraordinarily flexible logic engine based on the concepts of “behaviors” and “bindings”. You’ve already seen a “behavior” when we mentioned “subscribing” to alarm messages. That is an “alarm behavior”.

MapView documentation describes these concepts in detail. For now, you know just enough to understand the diagrams that follow.

What is AMPP?

AMPP® is a SaaS platform designed to enable efficient and flexible media production workflows for content creation, management, and distribution. It gives broadcasters and media companies a powerful, scalable, and secure platform to create and deliver more content to more channels as their facilities grow. Integration with functionality that exists in GV Orbit will form the orchestration layer of the Grass Valley Media Universe.

Once an AMPP platform has been deployed, you can access an ever-growing number of applications that may be started and stopped as necessary. The system is managed from a simple and secure unified user interface.

The AMPP platform is built on a microservice architecture where processes communicate over a network to meet a specific need. AMPP is built on over 100 small, decentralized, messaging-enabled, context-bound, individually deployable, services that combine to implement specific use cases like storing and retrieving data.

If you follow contemporary software trends, it won’t surprise you to learn that AMPP uses Kubernetes to coordinate all of the communication between microservices. If that is not familiar to you then all you need to know is that it’s the industry standard and it’s been around for nearly 20 years. In other words, the foundation is rock solid.

AMPP/GV Orbit Integration Concepts Summarized
The essential concept underlying the ability to use GV Orbit together with AMPP is that there are GV Orbit apps in the AMPP platform. AMPP Apps are one type of “workload” in the AMPP microservice ecosystem. GV Orbit apps can acquire data from AMPP and send it to GV Orbit. And GV Orbit can start (spin up) and stop (spin down) AMPP workloads. That integration will continue to develop and the lines between platforms will blur over time. But, even with a few initial integration concepts, fascinating solutions abound. Let’s look at two of them.

Example 1: A Penalty-Box

 

In even relatively small control rooms, and certainly in large master control rooms and playout facilities, the video walls fed by multiviewers contain far too many video displays to be monitored for quality and correctness by the human eye. Even were this possible, visually identifying PTP errors, incorrect multilanguage audio channel placement, and missing SCTE-35 data would be impossible.

Commonly operators in these facilities will benefit from what is known as “Monitoring by Exception”. This focuses operator attention only on errant signals. One way to implement this is to designate one or more locations on the monitor wall as areas known as a “penalty box”. This term can mean different things to different people. But, irrespective of how it’s defined, a flexible tool for accomplishing one of the myriad possibilities will bring laser focus to signals that are, in one way or another, misbehaving. MapView provides monitoring, control, and logical constructs to implement this sort of functionality using what are known as “behaviors” and “bindings”.

Behaviours and Bindings

From version 3.4 and beyond, GV Orbit began to include AMPP-related behaviours and bindings in MapView.

 AMPP Workload

This lets us create penalty-boxes which can look after both traditional hardware products and AMPP “workloads”. To do this, we use a mixture of the AMPP control mechanisms and the AMPP behaviors that have recently been added to Mapview, in several different ways.

Further, MapView now includes the first round of releases of new Kaleido behaviors.

 MapView

We can build an example penalty box using a combination of those.

Let’s look at the RTSP stream output of a Kaleido IP multiviewer This is similar to what you would see on your monitor wall, though in a greatly simplified form.

Monitoring by exception

Now let’s look at an example penalty box with an error.

Penalty Box error

In this example, you don’t necessarily need the multiviewer to be displayed on the right-hand side of the MapView screen since that will be what you see on the monitor wall in the room. But to illustrate how easy it is to include video streams in MapView, this example includes it.

There are, however, two things that you unquestionably will care about.

The monitor wall should include a designated penalty box (or multiple penalty boxes) for displaying signals with errors. In this example these sit on the right-hand side of the screen/wall where you are viewing the multiviewer output. These displays only appear and contain video when something about a video goes into error.

 Monitoring by exception penalty box

The second thing you should care about is a display that helps you rapidly diagnose an error. This is what you can see in the large window on the MapView UI.

Penalty Box channel 3

Suppose we lose synchronization on a Densitè XIP card because the PTP domain has changed. This is an error, and we need to highlight it. When lock is lost, after a few moments, the XIP card will realize this because it can't find anything on the non-existent PTP domain. In our example, GV Orbit will respond in 2 (among numerous possible) ways. First, an error will be shown at the bottom of the screen by virtue of MapView’s Monitoring by Exception widget.

Widget

Second, a flow diagram will open, illustrating the signal path and the location of the error.

 Channel 1 XIP-1

The Monitoring by Exception widget gets its alarm triggers from a favorites group that includes the devices we care about. This is just one way to do this but, in this case, the monitoring by exception group contains folders called Channel 1, Channel 2, and channel 3. Consider the first two. Channel one has only a single level below it. Channel two contains both Kaleido monitoring and AMPP playout.

The flow diagram is a tool that we created to zero in on the product within the group that has caused the error. We can see it in red and we can open its UI and fix the problem with a single click.

 Flow Diagram UI

That is an example of monitoring a modular card/frame in a traditional topology. But, with the new AMPP integration, we can take this a step further. Now we can monitor and respond to errors in an AMPP workload. To see that, consider creating alarms based on AMPP behaviors.

Let’s say that pausing a video will generate a warning; it does not mean the system has errored but we still want to know when it occurs. In out example, you will immediately see a warning in the GV Orbit tree view and then in the MapView display.

GV Orbit Tree

This is because the clip player knows it's been stopped MapView is configured to generate a warning.

As we see, a stopped player issues a warning because that is how we defined the behavior. Correspondingly, a frozen video is more serious. We have configured it to issue an error. So, after a few seconds the Kaleido IP, being another contributor to our alarm, now reports that error and, on the flow diagram, it turns red.

We have aggregated status information and then built some logic within MapView to determine what is an error. Then we push that error state to our diagram using the newly released Kaleido behaviors. We are also pushing an alarm state and some text using these Kaleido behaviors to the Kaleido IP.

These are but 2 of thousands of possible ways to integrate AMPP, Kaleido, and GV Orbit. But, building a penalty box is also only one of thousands of applications.

 

Read part 2