IP Video Engineering

Introduction

Two books in this series come before this one and both are about the network. One teaches you to configure the switch; the other teaches you to design what the switch sits in. This book is about the thing travelling through it, which turns out to be the part nobody covers.

That gap is real and it shows up on jobs. A video engineer who knows networking can tell you the link is at 40 per cent and the multicast table is correct, and still not be able to say why one screen is tearing and five are not. Answering that needs the other half: what the signal actually is, how it was cut into packets, what the codec threw away, how the receiver rebuilds the timing, and what each failure looks like on a screen.

So this book goes down rather than out. It takes resolution, frame rate, colour sampling and bit depth and shows how they become a bit rate. It takes the pixel group — the unit an uncompressed essence is built from — from the specification that defines it. It takes the codecs and says what each one trades. And it spends two chapters on artefacts and measurement, because reading a fault backwards from the screen is the skill that distinguishes an engineer from somebody who swaps boxes.

It assumes both networking books. Where a network setting matters, it is named and those books are cited rather than repeated.

How to Use This Book

Read it in order the first time. It follows the signal: what it is, how it is carried, what compression does to it, which standards wrap it, how its timing is kept, how it fails, and how to measure any of that.

Keep a notebook

Record every artefact you ever see, with three things beside it: what it looked like, what it turned out to be, and what measurement proved it. Artefacts repeat. Video faults are a small vocabulary of appearances with a small vocabulary of causes, and your own list of which went with which is the fastest diagnostic tool you will ever own.

A note on the words

Broadcast and AV use different words for the same things and this book is written across that line. An "essence" is a single stream of one kind of content — broadcast's word, and the right one once video, audio and data travel separately. A "feed" in AV is usually what broadcast calls a "source" or a "signal". "Sync" means timing reference to a broadcast engineer and lip-sync to most other people. And "uncompressed" in marketing often means lightly compressed, so the book says which it means and you should ask.

A note on safety

The equipment this book is about is heavy and is often in the air. An LED wall is a rigged, flown structure carrying real load, and its processing racks are mains-powered and get hot; a large display is a two-person lift at minimum and frequently a ladder job onto a bracket somebody else specified. Nothing in this book qualifies you to rig, suspend or fly anything, and the one thing worth knowing at this level is that a video engineer's weight and power figures feed somebody else's structural and electrical calculations, so getting them wrong is a safety matter rather than a paperwork one.

Two specific hazards belong to this discipline. Laser projectors emit beams that can injure an eye permanently, and the exclusion zones around them exist for that reason rather than for picture quality. And working behind a live video rack means mains power inches from your hands in poor light, which is where people actually get hurt — not on the network.

What IP Video Really Is

IP video is a video signal that has been cut into packets. Almost everything surprising about it follows from one fact: the cutting is lossless and the reassembly is not guaranteed, so a great deal of machinery exists to make the second part behave like the first.

Carrying video over networks.

What it replaced, and what carried over

It replaced SDI, and the comparison is worth drawing properly because the reader's instincts come from SDI and most of them still hold.

SDI is a constant-rate serial signal on coax. One cable carries one video signal, with its audio embedded inside it, at a fixed bit rate — 1.5 gigabits for HD, 3 gigabits for 1080p60, 12 gigabits for UHD. It either works or it does not, the failure mode is the whole picture going, and you can test it with a single instrument. Timing came separately, as black burst or tri-level sync on its own coax, from a sync pulse generator.

The parts that carried over: the signal is still a sequence of pictures at a fixed rate, the sampling and bit depth mean exactly what they meant, and the timing still has to come from somewhere. If you can read a waveform monitor, that skill is unchanged.

The parts that did not: the audio is no longer embedded, the timing now arrives as packets, and the failure modes are many rather than one. An SDI cable fails by going black. An IP essence can arrive late, arrive out of order, arrive with a gap, arrive with the wrong description of itself, or arrive perfectly at a receiver whose clock disagrees — and each of those looks different on the screen.

It is also worth knowing this is not the first attempt. Earlier RTP payload formats for video existed — RFC 2431 and RFC 3497, for MPEG-2 and for SMPTE 292M — and RFC 4175, the format everything now builds on, says explicitly how it relates to them. Both were narrower: tied to one compression scheme or one signal standard. RFC 4175's move was to describe the raw sampled picture in a general way, which is why it survived and became the foundation of ST 2110.

The three things that have to be true

For an IP video stream to work, three conditions have to hold at once, and nearly every fault in this book is one of them failing.

The packets have to arrive. All of them, because an uncompressed essence has no redundancy — a lost packet is a lost piece of picture and there is nothing to reconstruct it from.

They have to arrive in time. A receiver holds a small buffer and plays out at a fixed rate; a packet that arrives after its moment is worthless even though it arrived intact. This is why jitter matters more than average bandwidth and why a link at 40 per cent can still break video.

The receiver has to know what they are. An IP video stream does not describe itself the way SDI does. The receiver has to be told the resolution, the frame rate, the sampling, the depth and the packing, and if it is told wrong it will decode confidently into nonsense. That description is usually an SDP file, and a whole chapter is about it.

Hold those three in mind and the rest of the book is their machinery: RTP numbers the packets so arrival can be checked, timestamps let the receiver judge lateness, buffers absorb jitter, and SDP carries the description.

How to do it

1. Get the resolution, frame rate and scan from whoever owns the source.
2. Get the colour sampling and the bit depth, which decide the bit rate.
3. Establish whether it is uncompressed or compressed, and if compressed with what.
4. Find its multicast group address and its SDP file or equivalent description.
5. Work out the bit rate from the numbers, and check it against the documentation.
6. Establish what provides the timing, and whether the receiver follows the same thing.

What goes wrong

Practice it

On any IP video stream you can get documentation for.

The Video Signal Before It Is Packets

Six numbers describe a video signal, and every bit rate, every codec choice and most artefacts in this book follow from them. An engineer who cannot state all six for the signal in front of them is guessing.

Resolution, frame rate and scan

Resolution is the pixel count: 1920 by 1080, 3840 by 2160. Frame rate is how many of those pictures per second. Scan is whether each picture is complete — progressive — or split into two interlaced fields sent in turn.

Interlace is legacy and still very much present, and it matters here for a specific reason: an interlaced signal packs differently, and the specification for uncompressed video says so explicitly. A receiver told progressive when it is being sent interlaced produces a picture with combing on every moving edge, and that is a description fault rather than a signal fault.

Frame rate carries one trap worth naming. 59.94 and 60 are not the same, nor are 29.97 and 30, and the difference is a legacy of American colour television that will outlive everybody reading this. A plant running at 59.94 with one device at 60 will drop or repeat a frame periodically — about once every seventeen seconds — and the symptom is a judder somebody will swear is a network problem.

Colour sampling

A pixel's colour is carried as one luminance value and two chrominance values. Because the eye is far more sensitive to brightness detail than to colour detail, the chrominance can be sampled less often, and that is the single biggest lever on bit rate there is.

4:4:4 samples chrominance for every pixel. Nothing is discarded. Used for graphics, computer sources and anything that will be keyed heavily.

4:2:2 samples chrominance for every second pixel horizontally. This is the production standard, and it is the right default for almost everything in this book. The visible cost is almost nothing on camera images.

4:2:0 samples chrominance for every second pixel both horizontally and vertically, so chrominance values are shared between adjacent lines. This is what distribution uses — broadcast to the home, streaming, every consumer format. It halves the chroma data again and it is where chroma edges start to look soft.

4:1:1 samples chrominance every fourth pixel horizontally. Older, and you will meet it in legacy material rather than in a live plant.

The practical rule: shoot and move things around the plant at 4:2:2 or better, and only drop to 4:2:0 at the point of distribution. Chroma subsampling is not recoverable, so each step down is permanent.

Bit depth, colour space and the rest

Bit depth is how many bits each sample gets. Eight bits gives 256 levels per component, which is not enough for a gradient: a clear sky or a lighting wash will show banding, visible steps where there should be a smooth ramp. Ten bits gives 1024 levels and the banding goes away. Ten-bit is the production floor and has been for years, and when somebody reports "compression artefacts" on a gradient, eight-bit depth is the first thing to check rather than the codec.

Colour space is what the numbers mean. Rec. 709 for HD, Rec. 2020 for UHD and wide gamut, and the sRGB-ish conventions that computer sources use. A signal carried correctly but interpreted in the wrong colour space looks washed out or oversaturated, and it is a description fault — the packets were fine. Carriage formats have a place to say which space is in use, and JPEG XS carriage has a dedicated colour specification box for exactly this.

HDR adds a transfer function — PQ or HLG — describing how code values map to brightness. The common failure is a PQ signal shown on a display expecting standard dynamic range, which looks flat and grey, or the reverse, which looks blown out. Again: not a packet problem.

RGB rather than YCbCr. Graphics and computer sources are often RGB, and some carriage formats support RGB and RGBA — with an alpha channel — directly. This matters because an RGB source converted to 4:2:0 somewhere in the chain will show coloured fringing on text, and that is one of the most common and most misdiagnosed faults in corporate AV.

How to do it

1. Read the resolution and frame rate off the source device, not off the brief.
2. Confirm progressive or interlaced, and check what the receiver has been told.
3. Find the colour sampling, and ask where in the chain it last changed.
4. Find the bit depth, and if it is eight, note it as a finding.
5. Establish the colour space and whether anything in the chain is converting it.
6. If HDR is involved, establish the transfer function and that every display agrees.

What goes wrong

Practice it

On any video source you can inspect.

RTP, and How Timing Survives

UDP guarantees nothing: not arrival, not order, not timing. Every IP video standard in use rides on RTP, which adds just enough on top of UDP to make a real-time signal reconstructable. Three fields do nearly all of it.

Encoding, the network, and decoding.

What RTP adds

RTP is specified in RFC 3550 and it sits directly on UDP. It does not retransmit, it does not guarantee delivery, and it does not slow down — all of which would be wrong for real-time media. What it does is label every packet well enough that the receiver can work out what happened.

The sequence number increments by one per packet. A receiver that sees a gap knows precisely how many packets were lost and where, which is how every loss statistic you will ever read is produced.

The timestamp says when this packet's content belongs in time — not when it was sent. This is the field that makes real-time media work: the receiver plays out according to the timestamp, so packets that arrived early wait and the output rate stays constant regardless of how unevenly the packets turned up.

The SSRC identifies the sender, uniquely, within the session. Two sources on one multicast group are distinguishable because of it, and a changing SSRC is how you know a source restarted.

A handful of other fields matter in practice: the marker bit, which in video conventionally marks the last packet of a frame; the payload type, which says what the content is; and the extension mechanism, which is where profiles add their own fields.

The timestamp, and the buffer

Put the timestamp and a buffer together and you have the whole of how timing survives an unreliable network.

The receiver holds packets briefly rather than playing them the moment they arrive. The buffer absorbs jitter — the variation in arrival time — so that a packet which took three milliseconds longer than its neighbour still makes its slot. Then it plays out strictly by timestamp, at the signal's own rate.

Which gives the dial every IP video engineer ends up setting: buffer depth is latency traded for tolerance. A deep buffer survives a nasty network and adds delay. A shallow buffer is responsive and breaks up the moment arrival timing wobbles. There is no correct setting, only a correct setting for a given network and a given latency requirement, and knowing that the dial exists is most of the skill.

A packet that arrives after its playout moment is discarded. It is worth sitting with that: the packet was intact, the network delivered it, no bandwidth was exceeded, and it was useless. This is why measuring average utilisation tells you so little, and why jitter gets its own section later.

RTCP, the channel nobody reads

Alongside every RTP stream runs RTCP, a low-rate control channel that reports on the session, and it is startling how rarely anyone looks at it.

Senders emit sender reports saying how much they have sent and carrying the mapping between their RTP timestamps and real time. Receivers emit receiver reports saying what fraction of packets they lost, the highest sequence number they saw, and the jitter they measured. RFC 3550 includes guidance on analysing these, and the reports are per-receiver, which is the valuable part: on a multicast video system the receiver reports tell you that five decoders are clean and one is losing two per cent, which localises a fault in seconds.

RTCP also carries the source description — including the CNAME, a stable identifier for an endpoint across restarts — and a BYE packet when a source leaves properly, which is how a receiver can tell a clean departure from a crash.

The transmission interval is deliberately kept to a small fraction of the session bandwidth so the control channel never competes with the media, which is why RTCP is cheap to leave enabled and worth asking your equipment to expose.

RTP as specified, and which parts you will use

The structure of RFC 3550, the RTP specification. The right-hand column is what each part is for in a video plant. Nothing here is optional reading in the sense that the protocol is already running in every stream you work with — the question is only whether you can read it.

The two rows to act on are the receiver reports and the translators. Receiver reports are already being generated by your equipment and they answer, per decoder, the question you most often need answered: is this fault at the source, in the network, or at this one receiver. And the translator entry is worth knowing because it reframes a familiar box: an SDI-to-IP gateway is, in RTP's own terms, a translator, and the specification's discussion of how translators must handle RTCP is why a poorly implemented gateway breaks reporting for everything downstream of it.

How to do it

1. Find where your equipment exposes RTP statistics; most do and most people never look.
2. Read the sequence-number gap count, which is your packet loss.
3. Read the measured jitter, which is arrival-time variation rather than loss.
4. If several receivers share a source, compare their reports against each other.
5. A fault on one receiver and not the others is local to that receiver or its path.
6. A fault on all receivers alike is at the source or on the shared path.

What goes wrong

Practice it

On any IP video system with statistics you can read.

Uncompressed Video on a Wire

An uncompressed essence is the picture itself, cut into packets along its scan lines. The unit it is cut into is the pixel group, and understanding it is what turns the bandwidth arithmetic from a rule into something you can derive.

Scan lines, fragments and the MTU

Uncompressed video is packetised by scan line. A packet carries one complete line, a fragment of a line, or several lines and fragments — whichever fits. The constraint is the path MTU: the specification says scan lines should be fragmented so the resulting packet comes in under it, because a packet that needs fragmenting at the IP layer is a packet whose loss costs you more than it should.

Work it through for HD and the reason becomes obvious. A 1920-pixel line at 4:2:2 and ten bits is 1920 times 20 bits, which is 4800 octets — well over a standard 1500-octet MTU, so every line is sent as several packets. At UHD it is four times that. This is why an IP video plant very often runs jumbo frames, and why the MTU is a number worth knowing rather than accepting.

One detail with a visible consequence: a packet must contain a whole number of pixel groups. Where a line's length does not divide evenly into pixel groups, the sender fills the remaining bits of the last group with zero and the receiver ignores them — the specification describes this as the trailing edge being black-filled to a group boundary. It is harmless, and it is the sort of detail that explains an oddity at the very edge of a picture that would otherwise look like a fault.

The pixel group

A pixel group — pgroup — is the smallest number of adjacent pixels that packs into a whole number of octets. It is not a fixed size: it depends on the sampling and the bit depth, and the reason is simple arithmetic. At ten bits per sample you cannot fit one pixel into whole octets, so you take enough adjacent pixels that the total comes out even.

This is where the book's bandwidth arithmetic stops being a rule of thumb and becomes derivable. Look up 4:2:2 at ten bits in the table below: five octets for two pixels. Five octets is forty bits, over two pixels, which is twenty bits per pixel — exactly the figure the bandwidth calculations in this series use for 4:2:2 ten-bit video. The arithmetic and the specification agree, and now you can work out the rate for any format rather than remembering one.

Working out any bit rate

With the pixel group you can derive the bit rate of any uncompressed essence in two steps. Divide the group size in octets by the pixel count to get octets per pixel, multiply by eight for bits per pixel, then multiply by the pixel count and the frame rate.

For 1080p60 at 4:2:2 ten-bit: five octets over two pixels is 2.5 octets per pixel, so twenty bits. Then 1920 times 1080 times 60 times 20 is about 2.49 gigabits per second. For 2160p60 at the same sampling: 3840 times 2160 times 60 times 20 is about 9.95 gigabits.

For 4:4:4 at ten bits the group is fifteen octets over four pixels, which is 3.75 octets or thirty bits per pixel — half again as much as 4:2:2, which is the real cost of full chrominance and the reason graphics feeds are so much more expensive to carry than camera feeds.

Add a little for packet headers — the figures above are payload — and you have a number you can take to a design meeting and defend, which is the point.

The pixel group, by sampling and bit depth

Pixel-group sizes from Section 4.3 of RFC 4175, the RTP payload format for uncompressed video — the format SMPTE ST 2110-20 builds on. Each cell gives the group size in octets and the number of adjacent pixels that group covers.

Three things are worth reading off this table. First, the derivation above: 4:2:2 at ten bits is five octets per two pixels, so twenty bits per pixel, which is where the 1920 x 1080 x 60 x 20 arithmetic comes from. Second, 4:2:2 is uniquely tidy — two pixels per group at every depth — which is part of why it became the production standard. Third, the 4:2:0 rows are the only place scan lines interact: because chrominance is shared between adjacent lines, the progressive group is a two-by-two block of pixels rather than a run along one line, which is why progressive 4:2:0 packetisation carries samples from two consecutive lines in each packet. Interlaced 4:2:0 cannot do that, so it sends two-pixel groups along each line and carries the colour samples only on every other line.

How to do it

1. Establish the sampling and bit depth.
2. Look up the pixel group size and its pixel count.
3. Divide octets by pixels and multiply by eight to get bits per pixel.
4. Multiply by horizontal and vertical resolution and by the frame rate.
5. Add an allowance for packet headers, and state that you did.
6. Check the result against the manufacturer's figure and investigate any disagreement.

What goes wrong

Practice it

With the table above and a calculator.

Compression — What Each Codec Trades

One distinction sorts every codec you will meet, and it is not technical: is this codec meant to move pictures around a plant, or to deliver them to somebody at the end? The answer decides everything else.

Mezzanine against distribution

A mezzanine codec exists to reduce a signal enough to carry it conveniently while leaving it good enough to be processed again — keyed, graded, cut, re-encoded — without visible damage. Its priorities are low latency, visual transparency, and surviving repeated encode and decode cycles. It uses intra-frame compression only: every frame stands alone.

A distribution codec exists to deliver a finished picture to a viewer in as little bandwidth as possible. Its priorities are compression ratio and nothing else much. It uses inter-frame compression: most frames are described as differences from their neighbours, which is enormously more efficient and makes the signal fragile and slow.

Nearly every bad codec decision in AV is a distribution codec used where a mezzanine codec belonged. It is an understandable mistake because the distribution codec's numbers look wonderful — a tenth of the bandwidth — and the cost lands somewhere that looks like a different problem: latency blamed on the network, artefacts blamed on the screen, a signal that falls apart after one more re-encode blamed on the editor.

So the question to ask of any codec proposal is simply whether this picture is finished. If anything downstream will process it, you need a mezzanine codec, whatever the bandwidth saving on offer.

Why inter-frame compression costs latency

Inter-frame codecs — H.264, H.265, and everything built for streaming — group frames. A group of pictures starts with a complete frame and follows it with frames described as changes. Some of those reference frames that come after them in time, which means the encoder cannot emit a frame until it has seen frames that follow it.

That is where the latency comes from, and it is structural rather than a quality of implementation. A long group of pictures gives excellent compression and costs tens to hundreds of milliseconds at the encoder before a packet has even been sent. No network improvement touches it.

It also explains the characteristic failure. Lose data in an inter-frame stream and the damage persists: every subsequent frame that references the damaged one inherits the damage, so a single loss becomes a smear that sits on screen until the next complete frame arrives. That is the blocky, sticky, spreading artefact everybody recognises from bad streaming, and it is a property of the codec rather than of the network that dropped one packet.

Intra-frame codecs lose a piece of one frame and recover completely on the next. The artefact is a brief, localised disturbance. Being able to tell those two appearances apart tells you which family of codec you are looking at before you have asked anybody anything.

The ones you will actually meet

JPEG XS is the current production answer: intra-only, designed for very low latency — a fraction of a frame — and visually lossless at moderate ratios. It has its own chapter next because it is what new plants use.

JPEG 2000 is the previous generation of the same idea. Intra-only, visually lossless, but with latency of a frame or more and considerable processing cost. It is still in service on contribution circuits and in archives, and you will meet it in buildings fitted out a decade ago rather than in new designs.

H.264 remains the most widely deployed video codec in the world and is correct for streaming, recording and anything a viewer watches. H.265 does the same job in roughly half the bandwidth at higher processing cost. Both are wrong for production paths and both are constantly specified for them.

Proprietary low-latency codecs inside AV-over-IP products are mostly intra-only schemes aimed at the same place as JPEG XS. Treat them as mezzanine codecs and ask the four numbers: bit rate, latency, whether it is intra-only, and how many encode cycles it survives.

How to do it

1. Ask whether this picture is finished or will be processed again.
2. If it will be processed, require an intra-only codec and say why.
3. Get the latency figure for the encoder and the decoder separately.
4. Ask how many encode-decode cycles the codec survives visibly intact.
5. Get the bit rate at the quality you actually need, not at its best case.
6. Write the answer down with its conditions, because the figures will be quoted without them.

What goes wrong

Practice it

For any video chain you can inspect.

JPEG XS and Low-Latency Carriage

JPEG XS is what production IP video has settled on, and it is worth a chapter because the reader will be handed it by name and because how it is carried explains what it can and cannot do.

What JPEG XS is for

It is an intra-only, very low latency, visually lossless codec standardised as ISO/IEC 21122, designed explicitly as a mezzanine format. Its latency is a fraction of a frame — a few scan lines in some configurations — which puts it in a different category from everything before it. And it is computationally light enough to run in hardware at the edge of a camera or a screen.

The engineering consequence is that it makes uncompressed-grade workflows possible on sane infrastructure. An uncompressed 1080p60 feed is about 2.5 gigabits; the same feed at a modest JPEG XS ratio fits comfortably on a gigabit link with latency a production chain can live with. That trade is why new designs reach for it rather than for uncompressed and 25-gigabit switching.

It is carried over RTP by RFC 9134, and that specification is the subject of the rest of this chapter — not because you will implement it, but because its structure answers the practical questions.

How it is carried, and why that matters

A JPEG XS frame is a codestream, and the codestream is what gets packetised. Alongside it travel two descriptive structures the specification names explicitly: a video support box and a colour specification box. That second one is worth noticing, because it is where the colour space and transfer characteristics are declared — which is to say, the thing that goes wrong when a correctly carried picture looks washed out has a defined place to live, and a well-implemented chain carries it.

The specification also has a section on traffic shaping and delivery timing, and this is the most practically important part for anybody designing or commissioning a system. A compressed essence does not produce a constant stream of bits: a complex frame produces more data than a simple one, so an unshaped encoder emits bursts. A burst that exceeds what a link can carry instantaneously fills a switch queue and drops packets, on a link whose average utilisation looks perfectly healthy.

That is the explanation for one of the most baffling faults in compressed IP video: the system works all day and breaks up only on the shots with the most detail. It is not the network being inadequate; it is the encoder bursting and nothing shaping it. The remedy is the encoder's traffic shaping, which is why the specification devotes a section to it and why it is a question to ask of any product.

What RFC 9134 specifies for JPEG XS carriage

The structure of RFC 9134, the RTP payload format for ISO/IEC 21122 (JPEG XS). The right-hand column is what each part means on a job. Two rows are the practically important ones, and they are flagged.

The two flagged rows are where this chapter earns its place. Traffic shaping explains a fault that is otherwise inexplicable: a compressed system that is fine on simple pictures and breaks on complex ones, at average utilisation nowhere near the link's capacity. And the colour specification box explains that a washed-out or oversaturated picture in a correctly carried stream is a declaration problem with a defined place to look, not a mystery and not a display fault. Both are questions to ask a manufacturer, and both have answers.

How to do it

1. Establish the compression ratio or target bit rate, and the latency it gives.
2. Ask whether the encoder shapes its output traffic, and get the answer in writing.
3. Establish how the colour space is declared and whether the receiver reads the declaration.
4. Get the SDP or equivalent description and check every parameter against the source.
5. Commission with the most detailed content you can find, not with a test card.
6. Record the peak bit rate under that content, not the average.

What goes wrong

Practice it

On any compressed IP video path you can reach.

The Standards That Matter

Four names cover almost everything, and what matters about each is not what it is but what it demands. Here is each one as a list of requirements, which is how an engineer should hold them.

NDI, SMPTE 2110, and AVoIP.

ST 2110, and the essence model

SMPTE ST 2110 is the broadcast standard, and its defining move is to carry video, audio and ancillary data as separate streams rather than one multiplexed signal. Video goes in one essence, each audio group in another, ancillary data such as captions and timecode in a third, each with its own multicast address.

That is a genuine improvement — you can route audio without touching video, take one audio group to one destination, scale each independently — and it is the thing that most surprises people from SDI, where the audio was simply inside the picture. The practical consequences are immediate: the multicast group count multiplies, the address plan becomes real work, and a receiver needs to be told about several streams rather than one.

What ST 2110 demands: uncompressed or lightly compressed essences and therefore serious bandwidth; precise PTP, because the separate essences are reassembled by timestamp and nothing else; a description mechanism, in practice SDP, for every essence; and generally a network designed for it. Its video essence builds directly on the uncompressed payload format whose pixel groups this book has already tabulated.

When ST 2110 appears in a brief, the question is whether this is a broadcast facility with a broadcast budget. If yes, it is correct. If somebody liked the sound of it, a JPEG XS system does the job on a fraction of the infrastructure.

NDI, SDVoE, and transport over the open internet

NDI dominates production and streaming workflows, and what it demands is bandwidth and discovery rather than precision. It is tolerant of imperfect networks, it finds its own sources by name, and it runs on ordinary gigabit infrastructure. It is not frame-accurate across sources in the way ST 2110 is, and for a great many jobs that does not matter at all. Its faults are usually discovery faults, which means they are name-service faults.

SDVoE is a hardware approach aimed squarely at matrix replacement: very low latency, high quality, on 10-gigabit infrastructure, with the processing in a chipset rather than in software. What it demands is 10-gigabit switching and a network it largely expects to own.

SRT and RIST are a different category and worth knowing because the moment video leaves a managed network they are the answer. Both add error recovery and retransmission on top of unreliable transport, trading latency — typically a few hundred milliseconds of buffer — for the ability to survive real packet loss across the internet. They are for contribution and remote production, never for an in-plant production path, and confusing the two is how a system ends up with half a second of delay nobody can account for.

SDP, the file that configures a receiver

This surprises people and it is worth stating directly: on an ST 2110 system, a receiver is largely configured by a text file. The Session Description Protocol carries the multicast address, the payload type, and every parameter the receiver needs — resolution, frame rate, sampling, depth, the packing, the colour information, the PTP domain. The payload format specifications for both uncompressed video and JPEG XS each devote a section to how their parameters map into SDP.

Two things follow. A receiver told the wrong thing decodes confidently into nonsense rather than failing, which is why "the picture is garbage but the stream is arriving" is a description fault and not a network one. And the SDP file is a document you can read, diff against another, and keep — which makes it one of the most useful diagnostic artefacts in an IP video plant and one almost nobody archives.

Get the SDP for every essence during commissioning and keep it with the documentation. When a receiver misbehaves a year later, comparing the SDP it is using against the one that worked is often the entire investigation.

How to do it

1. Name the standard in use for each path, from documentation rather than from the brand.
2. For each, write down what it requires of bandwidth, clock and description.
3. Count the multicast groups it implies, including separate audio and data essences.
4. Find out how receivers are told what to expect, and get a copy of that description.
5. Establish whether anything is leaving a managed network, and if so what protects it.
6. Record all of it; this is the document somebody will need at three in the morning.

What goes wrong

Practice it

For a system you work on.

Timing, Genlock and PTP for Video

Video timing used to arrive on its own coax from a box in a rack, and one day it became packets on the same network as everything else. Nobody announced this clearly, and it is the source of a whole class of faults that get blamed on the network.

What genlock was for, and what replaced it

In an SDI plant, a sync pulse generator distributed a reference — black burst or tri-level sync — to every device on its own cable. Every camera, every mixer, every frame store locked its own timing to that reference, which is why you could cut between two cameras without a glitch: both were producing frames at the same instants.

Genlock solved one problem precisely. Two sources that are not locked to a common reference drift apart, and cutting between them means the switcher has to wait, or re-time, or show a disturbance at the cut.

On an IP plant, PTP replaces the sync generator. A grandmaster distributes time as packets, every device derives its frame timing from it, and the reassembly of separate essences by timestamp depends entirely on everybody agreeing what time it is. So PTP is doing two jobs at once: the old genlock job of keeping sources in step, and the new job of letting a receiver put video and audio back together.

Which means PTP failure in a video plant is more serious than in an audio one. Audio loses its clock and drifts audibly. Video loses its clock and essences stop reassembling, cuts show disturbances, and receivers start discarding packets they judge late — several symptoms at once, none obviously pointing at time.

What a video engineer needs to settle

You will not configure PTP — that is the network design, and AV-over-IP & Network Design covers grandmaster nomination, boundary clocks and domains. What a video engineer must know is narrower and it is operational.

Know which device is grandmaster and what happens when it goes. Know the PTP domain number, because a device on the wrong domain works perfectly and is locked to nothing you care about. Know where every device reports its clock status, and look at it before a show rather than during one. And know the epoch and offset conventions in use, because video timing is referenced to a defined origin and a device with the wrong offset is a fixed number of frames out — which looks like a lip-sync fault and is a timing configuration fault.

Then know what still needs genlock. Plants are hybrid for years, and any SDI equipment in the chain still wants its coax reference. The gateway between the two worlds has to be locked to both, and a plant where the PTP grandmaster and the sync pulse generator are not themselves locked together is a plant with two times in it. That is a real and common fault, and it produces a slow drift between the SDI and IP halves that nobody can localise because each half is internally perfect.

How to do it

1. Identify the grandmaster and confirm every device is following it.
2. Confirm every device is on the same PTP domain number.
3. Read each device's clock status and lock state, and note anything not locked.
4. Confirm the offset or epoch convention is consistent across devices.
5. If the plant has SDI, confirm the sync generator and the grandmaster are locked together.
6. Cut between two sources and watch the transition; a disturbance is a timing symptom.

What goes wrong

Practice it

On any IP video plant you can inspect.

Artefacts, and Reading Them Backwards

What you can see narrows the cause faster than any instrument. Video faults are a small vocabulary of appearances, each with a short list of causes, and learning to read backwards from the screen is the most valuable diagnostic skill in this book.

The appearance tells you the family

Before reaching for a tool, look at the picture and ask what kind of damage it is. The answer almost always eliminates most of the possibilities.

A brief localised disturbance that clears on the next frame is packet loss in an intra-only or uncompressed essence. One frame lost a piece and the next frame is whole.

Blocky damage that smears and persists for up to a second is packet loss in an inter-frame compressed stream. The codec is propagating the damage until its next complete frame.

Horizontal tearing, a displaced band across the picture is timing: the receiver has output part of one frame and part of another, which means it is not being driven by a stable reference.

Freezing, then jumping is a receiver running out of buffered content and then catching up. The network delivered late, not wrong.

Banding on gradients is bit depth, not compression.

Coloured fringing on text and sharp edges is chroma subsampling somewhere it should not be.

Overall washed out, flat, or oversaturated is a colour space or transfer function mismatch. The packets were fine.

Confident, structured nonsense — the picture is clearly being decoded and is clearly wrong, perhaps skewed or in the wrong colours entirely — is a description fault. The receiver was told the wrong parameters.

Periodic judder at a regular interval is a frame rate mismatch, classically 59.94 against 60.

Packet loss and jitter are different faults

These two get conflated constantly and they call for different remedies, so it is worth separating them firmly.

Loss means packets did not arrive. Its causes are a full link, a full switch queue, a marginal cable, multicast not being forwarded correctly, or an unshaped encoder bursting. Its measurement is the sequence-number gap, which every RTP receiver already counts. Its remedy is upstream: more capacity, shaping, priority, or a better cable.

Jitter means packets arrived at uneven intervals, and the ones that arrived after their playout moment were discarded. Its causes are queuing on a busy path, a varying route, or clock instability. Its measurement is also in the RTP receiver reports, as a separate figure. Its remedy is either upstream — reduce the queuing — or local: a deeper receiver buffer, at the cost of latency.

The diagnostic value is this: a system with jitter and no loss measures as a perfectly healthy network, because nothing was dropped by the network. Everything was delivered. The receiver threw it away. If you only ever measure loss, that fault is invisible to you, and it is extremely common.

How to do it

1. Look at the picture and classify the damage before touching anything.
2. Say what family it is in — loss, timing, description, depth or colour.
3. Check whether other receivers of the same source show it too.
4. Read the loss and jitter figures separately for the affected receiver.
5. Only now choose a tool, and choose the one that tests your hypothesis.
6. Record the appearance, the cause and the measurement that proved it.

What goes wrong

Practice it

Build your own reference, deliberately, on a bench.

Test and Measurement

What separates an engineer from somebody who swaps boxes is measurement. Five instruments answer nearly every question in this book, and only one of them is a packet capture.

The five instruments

Your eyes, with known content. A test pattern with a grey ramp, a colour bar set, and some fine detail will tell you about depth, colour space and chroma handling in seconds. Carry known content and know what it should look like. This is the cheapest and most underrated instrument there is.

The receiver's own statistics. Loss and jitter, per receiver, already being counted. Learn where your equipment exposes them. Comparing two receivers of one source is the single most informative measurement in IP video, and it costs nothing.

The switch's port counters. Utilisation, errors and discards, per port. Errors point at layer one; discards point at queues and therefore at bursts. A port with discards and healthy average utilisation is the signature of an unshaped encoder.

A waveform monitor or analyser with IP input. The instrument that tells you what the signal actually is rather than what it is supposed to be: levels, gamut, timing relative to reference. If the plant is serious, there is one, and knowing how to drive it is a real skill.

A packet capture. The last resort and the definitive one. It settles whether the stream arrived, in what order, with what timing and with what parameters. Nothing else can end the argument between network and application.

Using a capture on video

Video captures are large and the trick is to capture narrowly and know what you are looking for before you start.

Filter to the one multicast group and capture for a few seconds, not a few minutes — a 2.5-gigabit essence is over 310 megabytes a second. Then ask three questions of it. Are the sequence numbers continuous, which answers loss definitively. Are the arrival times even, which answers jitter. And do the header fields and the payload parameters match what the receiver was told, which answers description faults.

Capture at the receiver, not at the source. A capture at the source proves the source is sending, which is rarely the question. The useful capture is taken where the fault appears, because it tells you what actually arrived.

And capture both a working case and a broken one if you possibly can. A capture of a fault is suggestive; a pair of captures that differ in exactly one respect is proof, and it is what you want in your hand when you tell a manufacturer their product has a problem.

How to do it

1. Put known content through the path and look at it.
2. Read loss and jitter at every receiver of the source, and compare them.
3. Read the switch port counters for errors and for discards, separately.
4. Form a hypothesis and write it down before capturing anything.
5. Capture at the receiver, filtered to the one group, for a few seconds.
6. Check sequence continuity, arrival timing, and parameters against the description.
7. Capture a working case for comparison if one exists.

What goes wrong

Practice it

On a bench or a quiet system.

SDI, IP and the Gateways Between

Every real facility is both SDI and IP for years, and the join between them is where a disproportionate share of faults live. It is also, in the specification's own terms, a thing with a name.

Video plus networking, and the realities.

What a gateway actually is

A box that takes SDI in and puts an IP essence out is doing something RTP has a word for. It is a translator: a device that takes a stream and re-emits it in a different form while preserving its identity. The specification distinguishes this from a mixer, which combines several streams into one with a new identity of its own, and it discusses at length how each must handle the control channel.

That matters practically. The specification requires a translator to forward and adjust RTCP so that reporting still works across it. A poorly implemented gateway that drops or mangles RTCP breaks reporting for everything downstream, which means your best diagnostic tool goes dark precisely at the boundary where you most need it. When receiver reports look wrong or missing on the IP side of a hybrid plant, the gateway is the first suspect.

Cascading makes it worse, and the specification says so about mixers: chain several and the identity and timing relationships get harder to follow at each stage. The design lesson is to minimise the number of conversions a signal passes through. Every SDI-to-IP-to-SDI round trip is latency, a re-timing, a potential re-encode, and a place where reporting can break.

Running a hybrid plant

Four things, learned the hard way by everybody who has done it.

One time. The sync pulse generator and the PTP grandmaster must be locked together, or the plant has two times in it and the halves drift apart. This is the single most important thing on the list.

Count the conversions. Draw the signal path and mark every SDI-to-IP and IP-to-SDI crossing. Each one is latency and risk. A signal that crosses four times has usually done so because the plant grew rather than because anybody chose it, and the drawing is what makes that visible.

Know the latency of each crossing. They are not free and they are not equal, and a chain of them is how a plant ends up with a lip-sync problem nobody can localise. Get a figure for each gateway and add them up.

Keep the vocabulary straight. This is the part that actually wastes time in meetings. Broadcast engineers have been doing this longer than anybody and their words are precise and different: essence, reference, timing, source. AV says feed, sync, delay, signal. When a broadcast engineer asks whether your sources are timed, they are not asking whether they are working. Learn their vocabulary rather than expecting them to learn yours — they have more experience of this problem than you do, and they are generally delighted to be asked.

How to do it

1. Draw the signal path from every source to every destination.
2. Mark every SDI-to-IP and IP-to-SDI crossing on the drawing.
3. Get a latency figure for each crossing and total them per path.
4. Confirm the sync generator and the PTP grandmaster are locked together.
5. Check that RTCP reporting survives each gateway.
6. Look for any path crossing more than twice, and ask why.

What goes wrong

Practice it

On a plant you have access to, or from its drawings.

Commissioning and Running a Video Plant

Proving a video plant is a different procedure from proving a network, and it has to be done after the network has been proved rather than instead of it. The order matters because it is what makes a fault attributable.

Prove each essence, then the path, then the show

The network books cover proving the network: cable, configuration, addressing, two laptops per VLAN. Assume that is done. What follows is the video commissioning, and it has three stages.

Each essence on its own. One source, one receiver, nothing else. Confirm the picture is right with known content, confirm the loss and jitter figures are zero and small, and archive the SDP. Do this for every essence — including the audio and data essences, which is where people stop early and regret it.

Each path end to end. Source through every gateway, processor and conversion to its real destination. Measure the latency of the whole path and write it down. This is where accumulated conversion latency becomes visible and where a lip-sync problem is cheap to fix.

The whole plant at once, on demanding content. Every source running, every destination subscribed, and — this is the part people skip — the most detailed, fastest-moving content available rather than a test card. A compressed plant commissioned on a static test pattern has been commissioned at the easiest moment it will ever face. Run it for longer than feels necessary, because clock problems and slow leaks only show over time.

Then record what normal looks like, because it is the handover document that actually gets used: per-essence bit rate under demanding content, per-path latency, loss and jitter at every receiver, port utilisation and discards, and clock status. Without that baseline nobody can tell an abnormal reading from an ordinary one.

Monitoring, and documenting a plant

A live plant should be watched for the things that are visible before they are visible on screen: port discards creeping up, jitter rising at one receiver, a clock that has re-elected, an essence whose bit rate has changed because somebody altered an encoder setting. All of those precede a visible fault, and all of them are free to watch if somebody has been shown where.

Documentation for a video plant is not the network documentation and the two are often confused. A video plant needs: the signal path drawing with every conversion marked; the essence list with each one's multicast address, format and bit rate; every SDP archived; the per-path latency figures; the clock arrangement including what is grandmaster and what is locked to what; and the baseline.

Add the artefact notebook if you keep one, and hand that over too. It is the least formal document in the pack and often the most useful, because it tells the next person what has already gone wrong here.

How to do it

1. Confirm the network has been proved first, and do not start before it has.
2. Prove each essence alone with known content, including audio and data essences.
3. Archive every SDP as you go.
4. Prove each path end to end and measure its total latency.
5. Run everything at once on the most demanding content you can find.
6. Record the baseline — bit rates, latencies, loss, jitter, discards, clock.
7. Write the six video documents and hand them over in person.

What goes wrong

Practice it

On the next plant you commission or can study.

IP Video as the Future

The engineering in this book is a chain: the signal's six numbers decide its bit rate, the bit rate and the latency requirement decide the codec, the codec decides the artefacts you will see, and the timing decides whether any of it reassembles. Read back along it and every symptom has a place it comes from.

The converged, network-based direction of video.

IP video understood

Here is the whole chain in one paragraph. Resolution, frame rate, scan, sampling, depth and colour space describe the signal. Sampling and depth give the pixel group, the pixel group gives bits per pixel, and with resolution and frame rate that gives the bit rate. The bit rate against the available link, with the latency requirement, chooses between uncompressed, mezzanine and distribution compression. The codec family determines what a lost packet looks like. RTP numbers and timestamps the packets so loss and lateness can be measured; the receiver's buffer trades latency for tolerance; PTP keeps every device agreeing on time so separate essences reassemble; and SDP tells each receiver what it is being sent. Every artefact in the artefacts chapter is one of those links failing.

The value of holding it as a chain is that symptoms become attributable. Banding is depth. Fringing is sampling. Smearing damage is an inter-frame codec. Tearing is timing. Confident nonsense is a description. Freeze-and-jump is lateness. None of those is "the network", and an engineer who can say which link failed is worth several who can only say that it broke.

It also tells you what to ask when somebody describes a fault over the phone. Not "what have you tried" but "what does it look like" — because the appearance narrows the cause before you have left the building.

Developing IP video expertise

Three things, in order of how much they pay back.

Build the artefact reference deliberately rather than waiting for it. On a bench, introduce faults on purpose — drop packets on an intra-only stream and on an inter-frame one, starve a buffer, feed a receiver the wrong parameters, run two sources at slightly different rates — and photograph each result. An afternoon of that is worth years of accidental exposure, and nothing else in this trade gives a better return for the time.

Learn to drive a waveform monitor with an IP input. It is the instrument that tells you what a signal is rather than what it claims to be, and the number of people in AV who can use one properly is small enough that it distinguishes you immediately.

Then go and talk to broadcast engineers, in their vocabulary. They have been doing packetised video longer than the AV trade has, their terminology is precise, and they are generally pleased to explain it. The scarcest and best-paid technical role in this trade right now is somebody who can hold both halves of this conversation, and the fastest route to it is spending time with the people who hold the older half.

Read the specifications themselves. They are free, they are shorter than you expect, and the three cited in this book — the uncompressed payload format, JPEG XS carriage, and RTP itself — contain the answer to most questions anybody will ask you about IP video.

Glossary

Practical sign-off

IP video is an emerging field. The learner understands IP video, how it works, the standards, and the converged skills it needs. A mentor signs off when the learner understands IP video engineering foundations.

• Explained IP video and why AV is moving to it
• Described how video is encoded, networked, and decoded
• Identified key IP video standards (NDI, SMPTE 2110, AVoIP)
• Recognised the converged video+networking skills IP video requires

How to do it

1. Ask what it looks like, before anything else.
2. Place the appearance in a family — loss, lateness, timing, description, depth or colour.
3. Name the link in the chain that family belongs to.
4. Check whether other receivers of the same source agree.
5. Measure the one thing that would confirm or refute it.
6. Record the appearance, the cause and the proof.

What goes wrong

Practice it

An honest audit.

Where to Go Next

This book is the video half of a three-book set. AV Networking Intermediate configures the network it runs on and AV-over-IP & Network Design designs it; if anything here made you want to change a switch setting or size a link, those are the two to go back to. Sideways from here, Live Audio takes the audio essence properly and Timecode & Sync takes timing distribution.

The first ninety days

Spend one afternoon on a bench breaking things on purpose, and photograph every result. Drop packets on an intra-only stream and on an inter-frame one. Starve a receiver's buffer. Feed a receiver the wrong sampling. Run two sources at 59.94 and 60. You will come away with an artefact reference it would otherwise take years to accumulate, and it is the single highest-value afternoon in this series.

Then derive the bit rate of every essence you work with from its pixel group rather than looking it up, until the arithmetic is automatic. And start the notebook: appearance, cause, proof. Three columns.

Where this leads

Broadcast engineering, IP video engineering in large venues and facilities, and the systems-integration roles where somebody has to own the whole signal chain. It is the scarcest technical skill in the trade at the moment and the shortage is not closing.

What the gear actually looks like

Every device named in this book, in one place. Recognition is the whole point of Chapter 4 — use this page to attach a picture to each name before you meet it on a dock at six in the morning.

Audio

Dynamic microphone

Condenser microphone

Wireless handheld

Radio rack

Mixing console

Stage box / snake

Power amplifier

Main loudspeaker

Subwoofer

Stage monitor wedge

In-ear monitors

DI box

Video

Camera

Video switcher

Scaler / processor

Projector

LED wall panel

Confidence monitor

Lighting & control

Conventional fixture

Moving-head fixture

Lighting console

Dimmer

Show controller

Media server

Networking & infrastructure

Network switch

Patch bay

Equipment rack

Power distribution

Cable tester

A properly coiled cable

Connectors, close up

XLR

TRS / TS ¼-inch

SDI on BNC

HDMI

Cat5e / Cat6 (RJ45)

Fiber optic

DMX (5-pin XLR)

AC power — not signal