AV Networking Intermediate

Introduction

There is a moment in this trade that everybody remembers. You have patched audio for years, you know what a bad cable sounds like, and one day the system arrives with a switch in the middle of it. The console talks to the stage box through that switch. So does the video. So does the control panel on the wall. And when it stops working, none of the things you know how to do are any help, because there is nothing to listen to and nothing to wiggle.

This book is about that switch. Not networking in general — there are better books for that, and one of them is cited throughout this one — but the specific, narrow, unglamorous set of things an AV technician has to be able to do to a network before a show will run through it. Build a VLAN. Read a subnet mask and know whether two devices can see each other. Turn on IGMP snooping and understand what you just did. Mark a stream so the switch protects it. Work out on paper whether a gigabit link is big enough. Prove the network empty, then put the AV on it.

It assumes Networking Basics: that you know what an IP address looks like, roughly what a switch is for, and that the internet involves packets. Everything past that is here.

How to Use This Book

Read it in order the first time. It is built the way a network is: the wire, then the switch, then the addresses, then the traffic AV puts on it, then the day — configure, commission, run, fix, hand over.

Keep a notebook

For every network you work on, write down the subnet, the VLAN numbers and what each one is for, the switch management address, and the one setting you had to change from factory default. That last column is the valuable one. Switch defaults break AV in a small number of repeated ways, and your own list of them will be faster than any manual.

A note on the words

IT and AV use different words for the same things, and the gap causes real trouble in real meetings. A "port" is a physical socket on a switch to an AV technician and a numbered service to a network engineer, and both meanings appear in this book — the text says which. A "stream" is one flow of audio or video to us and any TCP connection to them. "Latency" to us is milliseconds of delay you can hear; to them it is often the round-trip time of a ping. Say which you mean and the conversation gets easier immediately.

A note on safety

Networking looks like the safe end of this trade and it is not, because of where the work happens. The switch is in a rack, and a rack has mains power in it: a loose IEC lead or a plate removed to reach a cable sits inches from live terminals, and reaching behind a powered rack in the dark is how people get hurt. Ceiling-mounted switches and access points mean ladder work, often alone, often over seating. And switches are heavier than they look — a loaded chassis is a two-person lift into a rack above shoulder height, however briefly.

The hazard specific to networked audio deserves naming on its own. A clock fault, a wrong gain structure or a sample-rate mismatch can put full-scale digital noise into a PA with no warning and no fade, and anybody wearing in-ears or standing in front of a wedge when it happens is at real risk of hearing damage. Mute the outputs before you change anything in the clock or routing of a live audio network, and tell the people on stage what you are about to do.

Practically: a switch carrying a show is not a place to practise. Every exercise in this book is written for a bench switch or a quiet window, and says so each time. The one-change-at-a-time rule in the troubleshooting chapter is not a style preference; it is what stops a diagnostic session from becoming an outage. Nothing here qualifies you to do electrical work.

Why AV Runs on the Network Now

For most of this trade's history the cable was the system. One cable carried one signal from one box to one other box, and you could follow it with your hand. That is the thing that changed.

AV runs on networks — deeper than the basics.

What changed, and what did not

Audio, video and control have all moved onto the network, and they did it separately, for different reasons, over about twenty years. Audio went first because it is small: a channel of digital audio is about a megabit a second, and a gigabit link will carry hundreds of them, so replacing a 48-pair analogue snake with one cat cable was an obvious win. Control followed because control was already data. Video came last because video is enormous, and it only became practical when switches got cheap enough and codecs got good enough.

What did not change is the physics at either end. A microphone is still a microphone, a speaker still moves air, and a bad connector is still a bad connector. The network sits in the middle, and it is a shared resource rather than a signal path — which is the whole difficulty. A dedicated cable either works or does not. A shared network can work for everything except the one stream you care about, and it can do that intermittently, and the reason will be a setting rather than a fault.

So the skill this book teaches is not really networking. It is knowing which network settings matter to real-time media, and being able to check and change them.

The three things that now travel as packets

Audio. Dante is the one you will meet most, with AES67 as the interoperability layer between Dante and everything else, and AVB, CobraNet, EtherSound and Q-LAN still in service in installed systems. These are small, constant, and brutally sensitive to timing.

Video. NDI for production and contribution, SDVoE and the various proprietary AV-over-IP systems for matrix replacement, SMPTE ST 2110 for broadcast. These range from tens of megabits to several gigabits for a single feed, which is the widest spread of any traffic you will ever be asked to carry.

Control. Panels, processors, lighting over sACN or Art-Net, camera robotics, and every piece of monitoring software. Individually tiny. Collectively capable of filling a link with broadcast traffic if nobody segments it.

They have different needs and they often share one switch. That is the job.

How to do it

1. Find the switch or switches and photograph the front panel, including the model number.
2. Log in or ask for the configuration; if nobody can produce one, assume factory default.
3. List what is plugged in, by department, and which of them is real-time media.
4. Ask one question — is this network only AV, or does it carry the building too?
5. Find out who owns it. If the answer is the IT department, nothing changes without them.
6. Write down the management address of every switch before you need it in a hurry.

What goes wrong

Practice it

On any AV system you have access to that contains a network switch.

The Layers, and Which One Broke

The layer model is taught as seven names to memorise and is useless in that form. Its actual value is as a running order for fault-finding: start at the bottom, and do not move up until the layer below is proved.

The four layers you will actually use

There are seven layers in the formal model and you will use four of them. Layer 1 is the physical cable and the light or electricity in it. Layer 2 is the frame, the MAC address, and the switch — everything that happens inside one network segment. Layer 3 is the IP address and the router — everything about getting between segments. Layer 4 is TCP and UDP, the two ways of carrying a conversation.

The remaining three — session, presentation, application — matter to software developers and almost never to you. If somebody insists on all seven, agree politely and carry on using four.

The useful fact about layer 4 is that real-time AV is almost always UDP. TCP guarantees delivery by asking for anything missing again, which is exactly wrong for audio: a packet that arrives late is worse than useless, because the moment it was for has passed. UDP just sends and does not look back. This is why a network problem makes AV click and stutter rather than slow down, and it is why you cannot fix AV packet loss by waiting.

Working up the layers

The order is not negotiable, because every layer depends on the one under it. A link light is layer 1. An address conflict is layer 3. If you start at layer 3 on a layer 1 problem, everything you find will be a true fact that explains nothing.

The question to ask at each layer is the same: can I prove this layer is working, with something I can see? Link light and a cable test for layer 1. A MAC address appearing in the switch's table for layer 2. A successful ping for layer 3. The stream actually arriving, in a packet capture, for layer 4.

Most AV network faults, in practice, are layer 1 or a configuration setting at layer 2. Very few are the exotic thing you were worried about.

How to do it

1. Look at the link light on both ends. No light is layer 1 and nothing else matters yet.
2. Check the port statistics for errors and for the negotiated speed.
3. Look for the device's MAC address in the switch's address table.
4. Ping the device from the same subnet; then ping it from another.
5. If it pings and the media does not pass, capture packets at the receiver.
6. Only now consider the AV application's own settings.

What goes wrong

Practice it

On a bench switch with two devices and a laptop.

Switches, VLANs and Trunks

A switch is a machine that learns. Everything it does that surprises you — including every multicast problem in this book — follows from what it has learned and what it has been told to do with traffic it does not recognise.

Segmenting and managing AV networks.

What a switch is doing

When a frame arrives, the switch reads the sender's MAC address and writes down which port it came from. That is the MAC address table, and it is the whole of a switch's intelligence. Then it reads the destination address, looks it up, and sends the frame out of that one port. If the destination is not in the table, it has no choice but to send the frame out of every port, which is called flooding.

Flooding is not a fault. It is the switch's honest answer to a question it cannot answer. But it means that traffic nobody asked for arrives everywhere, and when the traffic in question is a 3-gigabit video stream, everywhere is a problem.

An unmanaged switch does only this and cannot be told to do anything else. A managed switch does the same thing but lets you configure it: VLANs, QoS, multicast handling, port speeds, and the energy-saving features that will break your audio. Every decision in this book requires a managed switch. If the budget bought unmanaged, the budget decided the design.

VLANs — separating the AV traffic

A VLAN makes one physical switch behave as several separate switches. Ports in VLAN 10 can talk to each other and cannot talk to ports in VLAN 20 at all, even though they are inches apart in the same box. Getting between them needs a router.

The reason to do it is the broadcast domain. Some traffic goes to everybody by design — address resolution, discovery, announcements — and in one flat network every device hears all of it. Add enough devices and that background chatter becomes a measurable load, and more importantly, a single misbehaving device can flood the entire network. A VLAN draws a wall around that.

For AV the standard shape is simple and you should be able to draw it from memory: one VLAN for audio, one for video, one for control, one for management, and everything else somewhere else entirely. On a small job you may collapse audio and control together. You should almost never collapse video in with anything, because video is the traffic that will swamp its neighbours.

Access ports, trunks and tagging

A port carrying one VLAN is an access port. You set it to VLAN 10 and anything you plug in is in VLAN 10, knowing nothing about it. This is how every AV device connects, because AV devices almost never speak VLAN themselves.

A port carrying several VLANs is a trunk port. It is how two switches join, and how one cable carries audio, video and control between them. On a trunk, each frame gets a small tag added saying which VLAN it belongs to, so the switch at the far end can put it back in the right place. The tag is defined by IEEE 802.1Q, which is where the phrase "dot1q trunk" comes from.

Trunks are where VLAN plans go wrong, and the failure has a signature: everything works within each switch and nothing works between them. The cause is nearly always a VLAN that exists on one switch but was never allowed on the trunk. Allowing it is one line of configuration, and forgetting it costs an hour.

One more thing on a trunk: the native or untagged VLAN. A trunk carries one VLAN without a tag, and if the two ends disagree about which one that is, traffic silently lands in the wrong VLAN. Set it explicitly at both ends and write it down.

How to do it

1. Read the existing configuration and save a copy off the switch before changing anything.
2. Create the VLAN with a number and a name that says what it is for.
3. Assign the access ports the AV devices will use, and label the patch panel to match.
4. If a second switch is involved, add the VLAN to the allowed list on the trunk at BOTH ends.
5. Set the untagged VLAN explicitly on the trunk at both ends.
6. Leave the management VLAN alone unless you are certain, because that is how you lose the switch.
7. Test with two devices in the VLAN before any AV gear is connected.

What goes wrong

Practice it

On a bench switch with two laptops. Not on a live system.

Addressing, Subnets and the Mask

Two AV devices will only talk to each other directly if they agree that they are on the same network. The mask is what decides that, and reading one at a glance is the most immediately useful skill in this book.

The address and the mask, read together

An IPv4 address is four numbers, each 0 to 255. On its own it tells you nothing about who it can talk to. The mask splits it into two parts: a network part, shared by everything on that segment, and a host part, unique to each device. Two devices can speak directly if and only if their network parts match.

A mask of 255.255.255.0 means the first three numbers are the network and the last is the host. So 192.168.10.40 and 192.168.10.95 are on the same network and will talk. 192.168.10.40 and 192.168.11.95 are not and will not — not without a router, no matter how short the cable between them is.

You will see the same thing written as /24, which is the count of bits in the network part. 255.255.255.0 is 24 bits, so /24. This is CIDR notation and it is how everybody writes it now, because it is shorter and because it handles the masks that do not land on a neat boundary.

How many devices fit

The host part decides the size. With 8 host bits you get 256 combinations, minus two that are reserved — the all-zeros network address and the all-ones broadcast address — so 254 usable addresses. That is the arithmetic, and it is worth being able to do it in your head for the three or four masks you will actually meet.

The practical lesson is to size up. A /24 feels enormous when you are commissioning 40 devices and feels tight the second time the client adds a wall of screens. Changing a mask later means re-addressing everything, during a window nobody wants to give you.

Static, DHCP and link-local

AV devices arrive in one of three addressing moods and you will meet all three on the same job.

Static means you type the address into the device and it keeps it. This is what installed AV should use, because a device whose address never changes is a device you can find at 3 a.m. Keep a written addressing plan — a spreadsheet is fine — because the alternative is discovering a conflict by watching two devices fight.

DHCP means a server hands out addresses from a pool. Convenient, and correct for laptops and guest devices. The trap for AV is that a device's address can change after a power cut, and anything configured to point at the old address stops working. If you must use DHCP for AV, use reservations so each device always gets the same address.

Link-local is what a device gives itself when it is set to DHCP and finds no server: an address in 169.254.x.x. This matters enormously in audio, because Dante devices default to it and a Dante system will happily run on link-local addresses with no DHCP server at all. It is also the thing to recognise when a device "has no network" — a 169.254 address means the device is fine and the DHCP server is absent.

One detail of link-local is worth knowing because it explains behaviour that otherwise looks like a fault. The scheme is specified in RFC 3927, and a device picking a link-local address does not just take one: it probes first, announces, and then DEFENDS the address with ARP if anything else claims it. So two devices cannot quietly end up on the same link-local address — one of them will give way and choose again. The timing is fixed rather than configurable, and the consequence for you is that a link-local system takes a few seconds to settle after power-up and should then be stable. A link-local address that keeps changing is a real fault, not the protocol working.

Finding each other by name

AV devices rarely ask you for the address of the thing they are talking to. They find each other, and the mechanism is usually mDNS — multicast DNS, the same thing that makes printers appear on a laptop. Dante Controller, NDI sources and most control systems all use some form of it.

Two consequences follow, and both cause real jobs to fail. First, discovery is multicast, so a switch that handles multicast badly breaks discovery even when the media itself would have worked. Second, discovery does not cross a router. Devices in different VLANs will not see each other in the browser list even though they could exchange packets perfectly well if you typed the address in by hand. That is not a fault; it is what a VLAN is for. When somebody wants discovery across VLANs, the answer is a specific piece of configuration on the router, and the honest thing to say is that it needs designing rather than switching on.

How to do it

1. Decide the subnet for each VLAN and write the CIDR, not just the mask.
2. Reserve the first few addresses for infrastructure — router, switches, servers.
3. Give each class of AV device a block, in order, with gaps for growth.
4. Leave a DHCP range at the top for laptops and anything temporary.
5. Count what you have assigned and check it against the usable total.
6. Put the plan somewhere the next technician will find it, not only on your laptop.

What goes wrong

Practice it

On paper first, then on a bench.

Multicast and IGMP Snooping

This chapter is the one that pays for the book. More AV-over-IP systems fail on multicast than on anything else, the failure is dramatic, and the cause is almost always one setting that nobody turned on.

One sender, many receivers

There are three ways to address traffic. Unicast is one sender to one receiver: the sender makes a separate copy for each. Broadcast is one sender to everybody on the segment. Multicast is one sender to a group of interested receivers, and only the interested ones.

For AV the difference is money. Say one camera feed is 200 megabits. By unicast, sending it to eight screens means the source transmits eight copies — 1.6 gigabits out of one port, which a gigabit link cannot do. By multicast the source transmits one copy, the switch duplicates it only on the ports that asked, and eight screens cost the same 200 megabits leaving the source as one does. That is why every serious AV-over-IP system uses multicast.

A multicast group is identified by an address in the range 224.0.0.0 to 239.255.255.255. Receivers do not connect to the sender; they ask the network to join a group, and the network's job is to get the traffic to them.

The address trap nobody warns you about

A multicast IP address gets mapped onto a multicast MAC address so the switch can handle it at layer 2, and the mapping only carries the low 23 bits of the IP address. The address has 28 bits that can vary. So 32 different multicast groups map to the same MAC address.

This is not academic. Two streams whose addresses differ only in a high bit — 239.1.1.1 and 239.129.1.1, for instance — look identical to the switch's layer-2 machinery, and a receiver that joins one gets both. On a system with many streams this shows up as mysterious extra load and occasional video appearing where it should not.

The practical defence is to keep every multicast address in a system inside one narrow block, and to let the AV system allocate addresses itself where it offers to. Most do, and they do it correctly.

IGMP, the querier, and snooping

IGMP is the protocol a receiver uses to say "I want group 239.1.1.5". It is defined in RFC 3376 for version 3, and the mechanism is a conversation: a router or switch acting as the querier sends out periodic General Queries asking who still wants what, and receivers answer with Membership Reports.

IGMP snooping is the switch listening in on that conversation. Without it, the switch sees multicast as traffic it does not recognise and floods it out of every port — which means your 3-gigabit video stream arrives at the control panel, the audio DSP and the laptop, and the network collapses. With snooping on, the switch builds a table of which ports asked for which group and forwards accordingly.

There is a version trap worth knowing precisely, because the usual advice is too vague to act on. RFC 4541, which is the recommendation for how snooping switches should behave, describes the specific case: an IGMPv3 router with an IGMPv2 snooping switch between it and the hosts. The router keeps running v3 even when the hosts are v2, so the network never settles on v2 — and a v2 snooping switch will not recognise the v3 membership reports it is seeing. The groups in those unrecognised reports then get either flooded or pruned, which is to say your video goes everywhere or nowhere.

The RFC's recommendation is concrete and it is the thing to ask for: configure the multicast router to use IGMPv2. If that is not possible and the switch cannot process v3 reports, the RFC says to disable the switch's snooping altogether, because there is no clean answer — which is a case where turning snooping OFF is the correct engineering decision rather than a failure. Knowing that exists stops you fighting a switch that was never going to win.

The part that catches people is that snooping needs a querier. If nothing on the segment is sending queries, receivers eventually stop reporting, the switch's table empties, and it goes back to flooding — or worse, stops forwarding altogether. On an AV-only network with no router there is often no querier at all, so you have to enable one on a switch. Exactly one. Two queriers on a segment elect a winner, and the election is one more thing to go wrong.

The IGMP timers, and why a stream keeps arriving

Defaults from Section 8 of RFC 3376, the IGMPv3 specification. These are configurable, and the RFC is explicit that if you change them they must be consistent on every system on the link — a mismatched timer is worse than a slow one.

Read the Group Membership Interval line again, because it explains a fault that gets blamed on hardware every week. Two times 125 plus 10 is 260 seconds. If a receiver disappears without sending a Leave — somebody pulled the cable, or the device crashed — the network can keep forwarding that stream for over four minutes. On a busy video system that is four minutes of bandwidth spent on nobody, and if several receivers went at once it is enough to break everything still running. The trade calls this leave latency, and a ghost stream nobody is watching is what it looks like from the outside. It is why Leave processing matters: a receiver that sends a Leave is dropped after the last-member queries, about two seconds at the defaults, while one that simply vanishes waits out the full Group Membership Interval. If streams must stop sooner after a crash or a pulled cable, it is the Query Interval that has to come down, on every system on the link.

How to do it

1. Enable IGMP snooping globally, and then per VLAN if the switch asks separately.
2. Find out whether anything on the segment is already a querier — a router usually is.
3. If nothing is, enable the querier on exactly one switch, and write down which.
4. Check the IGMP version; mixing versions on one segment causes the protocol to fall back.
5. Bring up one source and one receiver and confirm the stream arrives.
6. Confirm it does NOT arrive at a third port that never asked for it. That is the actual test.
7. Add receivers one at a time and watch the switch's multicast table grow.

What goes wrong

Practice it

On a bench switch with one multicast source and two receivers. Never on a live show network.

QoS — Deciding What Gets Dropped

Quality of Service does not create bandwidth. It decides who suffers when there is not enough. Readers who expect the first thing are disappointed; readers who understand the second thing design better networks.

What QoS actually does

A switch port has a queue. When traffic arrives faster than the port can send it, the queue fills, and when the queue is full the switch drops something. Without QoS, what it drops is whatever arrived when the queue was full — which might be your audio.

QoS gives the switch several queues and rules about which to serve first. Traffic is marked, the switch sorts marked traffic into the right queue, and the important queue is emptied before the others. When the link is congested, the unimportant traffic is what gets dropped.

Two things follow. First, QoS does nothing on an uncongested link — if there is capacity for everything, every queue drains and the markings never matter. Second, QoS is only as good as the weakest hop: marking traffic at the source is pointless if the switch in the middle ignores markings, which is the default on a great many switches. QoS is an end-to-end agreement or it is nothing.

So the honest summary is this. If your link is congested, QoS decides what survives, and you want that decision made deliberately. If your link is not congested, QoS is insurance. Either way it is worth configuring, and either way it is not a substitute for a big enough link.

Marking, and what the marks mean

The mark is six bits in the IP header called the DSCP — the Differentiated Services Code Point. Six bits gives 64 values, and the industry agreed on which values mean what so that equipment from different vendors can interoperate. That agreement is RFC 4594, and it is worth knowing that it exists, because it is the answer to "why that number and not another one".

The mechanism under the mark is the Per-Hop Behavior: what each switch along the way promises to do with traffic carrying that mark. Expedited Forwarding means serve it first, with a strict limit on how much of it you accept. Assured Forwarding means give it a guaranteed share with several drop priorities inside. Class Selector values are the older, simpler scheme that just says "more important than the one below". Default Forwarding is everything nobody marked.

Two of these matter most to AV, and they are worth separating. Audio and clock want to be served first, every time, with no exceptions — that is Expedited Forwarding territory. Video wants a large guaranteed share but can tolerate being queued behind a packet of clock, because one frame of video late is a glitch and one packet of clock late is a system-wide fault. Getting that priority order the right way round is the single most important QoS decision in AV, and it is the one people get backwards because video is the thing you can see.

The DSCP values, and which service class AV traffic belongs in

Figure 3 of RFC 4594, the IETF's Configuration Guidelines for DiffServ Service Classes. The RFC expects administrators to start with three or four classes and add more only as the need arises, so an AV network typically uses a handful of these rather than all twelve.

Three rows are the ones to recognise on an AV job. Broadcast Video at CS3 is where a live video feed belongs — the RFC names broadcast TV and live events explicitly. Real-Time Interactive at CS4 is for two-way video where delay is intolerable. Expedited Forwarding at EF is defined for telephony bearer traffic, which is the closest thing in the RFC to low-latency production audio, and it is why networked audio systems commonly mark their media EF and their clock higher still. What matters is less which exact value you pick than that every switch on the path agrees, and that clock is never below media.

How to do it

1. Find out what the AV equipment already marks its traffic as; most networked audio marks it for you.
2. Decide the order — clock first, then media, then control, then everything else.
3. Configure the switch to trust the markings coming in on AV ports.
4. Map each marking to a queue, with the clock and audio queue served strictly first.
5. Do the same on every switch on the path, because one that ignores markings undoes the rest.
6. Verify under load, not on a quiet network, because a quiet network cannot show you a difference.

What goes wrong

Practice it

On a bench with one audio stream and a way to generate bulk traffic.

Clock and Timing on the Network

Networked audio fails on clock long before it fails on bandwidth. A system with a tenth of the traffic it can carry will still be unusable if the clock is wrong, and this is the part of AV networking that has no analogue in general IT.

Why clock is a network problem now

Digital audio is a sequence of samples taken at a fixed rate, and playback has to use the same rate the recording used. In an analogue world this was handled by a word clock cable. Once audio travels as packets over a shared network, there is no clock cable — so the clock has to travel as packets too, and arrive accurately enough that devices kilometres of cable apart agree on when a sample happens, to within microseconds.

The protocol that does this is PTP, the Precision Time Protocol, IEEE 1588. One device is elected the grandmaster and everything else follows it. The election is automatic, based on advertised quality, and the important practical fact is that it happens whether you planned it or not: plug in a network of audio devices and one of them becomes the clock.

AES67 uses PTP version 2. Dante has historically used version 1 and moved to supporting version 2 for AES67 interoperability, which is why a mixed system can need both running and why interoperability questions so often turn out to be clock questions. When you join two networked audio systems together, the first thing to establish is which one is the clock and whether they are speaking the same version of PTP.

What you do about it

Three decisions, and then mostly you leave it alone.

First, choose the grandmaster deliberately. Let the system elect one and it will usually pick something sensible, but "usually" is not a design. Pick a device that is always powered, centrally placed in the network, and ideally one built for the job. Set its priority so it wins the election, and set a second device to be the next-best so there is a fallback.

Second, give the clock the highest priority on the network. Clock packets are tiny and infrequent and the cost of prioritising them is nothing. The cost of not doing it is a system that works until the network gets busy and then drifts.

Third, keep the path short and consistent. Every switch a clock packet crosses adds a little uncertainty, and a packet that takes a different route each time is worse than one that takes a long route every time. This is one of the reasons AV networks are built flat and simple rather than clever.

The symptom of clock trouble is distinctive once you have heard it: not clicks from lost packets but a slow ticking, a periodic artefact, or audio that drifts out of sync with video over minutes. If somebody describes that, stop looking at bandwidth.

How to do it

1. Open the control software and find which device is currently grandmaster.
2. Decide which device should be, on grounds of always-powered and central.
3. Set its clock priority so it wins, and nominate a second as backup.
4. Check every device agrees on the same PTP version.
5. Give clock traffic the top priority queue on every switch.
6. Watch the system for a few minutes and confirm no device is re-electing.

What goes wrong

Practice it

On any networked audio system with control software you can see.

Bandwidth and the Arithmetic

You can size an AV network on paper in two minutes, and the arithmetic is multiplication. Doing it before the truck arrives is what separates a design from a hope.

Audio, which is small

One channel of digital audio is the sample rate times the bit depth. At 48 kHz and 24 bits that is 48,000 times 24, which is 1,152,000 bits per second — call it 1.15 megabits for the payload. Packet headers add to that, and how much depends on how many samples you put in each packet: low-latency settings send smaller packets more often, so the overhead fraction goes up. A working rule is to allow around 1.5 megabits per channel and you will not be far wrong.

So 64 channels is roughly 96 megabits. A gigabit link carries that several times over without noticing. Even a 256-channel system is around 400 megabits, which fits.

The conclusion is worth saying plainly, because it corrects an instinct: networked audio almost never runs out of bandwidth. When networked audio fails it is clock, or multicast, or a switch setting — not capacity. Do the arithmetic once so you can stop suspecting it.

Video, which is not

Uncompressed video is the pixel count times the frame rate times the bits per pixel. For 1080p60 at 4:2:2 and 10 bits, that is 20 bits per pixel: 1920 times 1080 times 60 times 20 comes to about 2.49 gigabits per second. One feed. A gigabit link cannot carry even half of it.

For 2160p60 — UHD — at the same sampling, 3840 times 2160 times 60 times 20 is about 9.95 gigabits. This is why uncompressed IP video lives on 10-gigabit and 25-gigabit infrastructure, and why a conversation about ST 2110 is always also a conversation about a different class of switch.

Compression changes the picture entirely. Lightly compressed systems aimed at matrix replacement typically run in the hundreds of megabits, which fits on gigabit. Heavily compressed streaming runs in the single digits of megabits and will cross a normal corporate network. The trade is latency and quality: the lighter the compression, the lower the delay and the bigger the pipe.

Two warnings about this table. The compressed rows are deliberately vague because they depend entirely on the product, and the number you must use is the one in that product's documentation rather than anything here. And every figure is per feed — the arithmetic that actually matters is the total across the busiest link, which is the next thing.

Adding it up across the busiest link

Bandwidth faults do not happen at devices. They happen at the one link everything has to cross: the trunk between two switches, or the uplink out of a rack. The design question is always the same — what is the total of everything that must cross this one cable at the worst moment?

Work it out for the busiest link and then leave headroom. Running a link above about 70 per cent of its rated speed is where queuing delay starts to become visible on real-time traffic, and a link planned at 95 per cent is a link that fails the first time somebody adds a camera.

Multicast changes this arithmetic in one important way, which is worth being precise about. A multicast stream crosses a trunk once no matter how many receivers are on the far side. So the total across a trunk is the sum of the distinct streams that need to cross it, not the sum of what every receiver consumes. This is the single biggest reason AV-over-IP designs work at all, and the reason unicast designs stop scaling at about four receivers.

How to do it

1. Draw the network and mark every link that carries traffic between groups of devices.
2. For each link, list the distinct streams that must cross it in either direction.
3. Multiply out each stream from the manufacturer's figure, not from memory.
4. Total them per link and compare against 70 per cent of the link speed.
5. If any link fails, the answer is a faster link, a second link, or moving a device.
6. Write the totals down; you will be asked to justify the switch specification.

What goes wrong

Practice it

On paper, for a system you have actually worked on.

Configuring a Switch for AV

This is the chapter to work through with a switch in front of you. It is the whole job in one order, start to finish, and the order matters because several of the steps will lock you out if you take them early.

Multicast, QoS, bandwidth — and setting it up.

What each system asks of the network

Before you configure anything, know what the gear wants. Each of the common systems asks of the network a slightly different set of things, and the differences are what your configuration has to satisfy.

Dante asks for a clean layer-2 network, PTP clock with one grandmaster, QoS with clock above audio, and IGMP snooping if it is using multicast flows. It is tolerant of modest networks and intolerant of clock trouble.

AES67 asks for much the same but specifically PTP version 2, and it is the layer that lets Dante talk to other manufacturers' audio. Most AES67 problems are clock-version problems.

NDI asks mostly for bandwidth and discovery. It is more forgiving of network imperfection than the audio protocols and much hungrier, and its discovery is a name service, so it fails in discovery ways.

SMPTE ST 2110 asks for a great deal: 10-gigabit or faster infrastructure, precise PTP, and usually a network designed for it from the start. If somebody says ST 2110, the network is a project rather than a setting.

Control traffic — sACN, Art-Net, panels, processors — asks for almost nothing except to be kept away from everything else, because some of it broadcasts cheerfully and constantly.

The settings that break AV

A managed switch out of the box is configured for an office. Several of those defaults are actively hostile to real-time media, and knowing this short list will save you more time than anything else in the book.

Energy saving. Features marketed as green Ethernet or EEE power down parts of a link when it looks idle and wake them when traffic arrives. For bursty office traffic this is fine. For audio it inserts small unpredictable delays, and the result is clicks nobody can account for. Turn energy-saving and EEE off on every AV port.

Spanning tree. STP stops loops, which you want, but when a link changes state the default timers can block traffic for tens of seconds while the topology recalculates. On an AV network use the rapid variant, and set ports that connect to AV devices as edge ports so they come up immediately rather than waiting.

Storm control and rate limiting. These cap how much broadcast or multicast traffic a port will pass, to protect the network from a faulty device. Set too low — and the defaults often are, relative to AV — they throttle your video. Either raise the threshold on AV ports with a number you calculated, or turn it off on ports you control.

Flow control. A port asking a sender to pause sounds helpful and is wrong for real-time media, where a paused packet is a late packet. Leave it off and solve congestion with QoS and capacity instead.

Automatic power-saving on the switch itself, fan profiles that throttle, and firmware defaults that re-enable things on reboot are all worth checking. Write down what you changed, because the next firmware update may quietly put some of it back.

How to do it

1. Save the existing configuration off the switch before anything else.
2. Set the management address and, if you use one, the management VLAN — and confirm you can still reach it.
3. Update firmware now if it needs it, because doing it later undoes nothing but costs a reboot mid-commissioning.
4. Create the VLANs, with names.
5. Assign access ports to VLANs and label the patch panel to match.
6. Configure trunks, with the allowed VLAN list and the untagged VLAN set explicitly at both ends.
7. Turn off energy saving and EEE on every AV port.
8. Set rapid spanning tree and mark AV device ports as edge ports.
9. Raise or disable storm control on AV ports, and leave flow control off.
10. Enable IGMP snooping per VLAN and a querier on exactly one switch.
11. Configure QoS — trust markings on AV ports and set the queue order with clock first.
12. Save the configuration, then export a copy off the switch and put it somewhere findable.

What goes wrong

Practice it

On a bench switch you are free to break.

Commissioning the Network

A network proved empty is a network you can trust when it is full. The temptation is always to plug in the AV and see if it works; the discipline is to prove the network first, because then every fault that follows belongs to the AV.

Run and fix AV networks.

Prove the network before the gear

Commissioning has an order, and the order exists so that faults are found one category at a time. Prove the cabling, then the switch configuration, then the addressing, then put one AV device on, then two, then the rest. Each stage produces evidence you write down.

The reason is simple arithmetic about fault-finding. If you connect forty devices to an unproved network and something is wrong, you have forty-one suspects. If you proved the network first, you have one.

Prove the cabling with a tester, not with a link light — a light tells you two pairs work and says nothing about the other two or about crosstalk. Prove the switch configuration by reading it back rather than by remembering what you typed. Prove the addressing with two laptops before any AV device is involved. Only then start connecting gear.

Connecting the AV devices

Work from the addressing plan, and work in one order: physical connection, then address, then onto the right VLAN, then into the control software, then into the show. Do one device completely before starting the next, because half-configured devices are what make a system impossible to diagnose.

Expect the gear to disagree with the plan. A device will be on an old address from a previous job, or set to DHCP when the plan says static, or still bound to another system's clock. This is normal and it is why the plan is written down — so that the disagreement is visible rather than absorbed.

As each device comes up, note three things: its address, which switch port it is on, and whether anything had to be changed from factory default. That last note is what makes the system maintainable by somebody who is not you.

Monitoring a live network

Once the show is running, the useful question is not "is it working" but "is anything getting worse". A link that is quietly accumulating errors, a utilisation figure that climbs every time a department wakes up, a device that has re-elected its clock twice — these are the faults you can still fix.

Watch the link utilisation on the busiest trunk, the error counters on every AV port, and the clock status. Most managed switches will show all three, and most AV control software shows the last one. Look at them during rehearsal, not for the first time during the show.

Write down what normal looks like. A trunk at 30 per cent and two ports with a handful of errors is a fact, not a problem — but you can only know that if you recorded it when everything was fine. Half of monitoring is having a baseline.

How to do it

1. Test every cable with a tester and record the results.
2. Read the switch configuration back and compare it against what you intended.
3. Put two laptops on each VLAN and confirm they talk within it and not across it.
4. Connect one AV device, address it, and prove it end to end.
5. Connect a second and prove the two together, including any multicast stream.
6. Connect the rest, one at a time, noting address and port for each.
7. Run the whole system at full load and record utilisation, errors and clock status as your baseline.

What goes wrong

Practice it

On the next system you commission, however small.

Troubleshooting Networked AV

Under pressure, method beats knowledge. The technician who works one change at a time through a known order will beat the one who knows more and guesses, every time, and the difference is widest when it matters most.

The method

Start by establishing what is actually broken, in terms somebody else could check. "Audio is down" is not a symptom; "channels 1 to 16 from the stage box are silent at the console, and channels 17 to 32 from the same box are fine" is. The second one has already halved the problem.

Then halve the system. Half the suspects go, you test, and whichever half contains the fault gets halved again. Four or five rounds of this will isolate almost anything, and it works without any knowledge of what the fault is.

Change one thing at a time and put it back if it did not help. Two changes at once means you no longer know what fixed it, and a session of accumulated un-reverted changes is how a fault becomes an outage. Write down what you changed. You will not remember at minute forty.

And work up the layers. A bad patch lead presents as a software problem with impressive frequency, and the order in the layers chapter exists precisely so that you spend your first two minutes on layer one rather than your last two.

The five tools

A cable tester. Not a link light. The thing that tells you all four pairs are good and correctly ordered. The highest hit-rate tool in the bag.

Ping. Proves layer 3 both ways. Ping from the device as well as to it if you can, because one-way reachability is a real and confusing condition. A ping that succeeds tells you addressing and routing are fine, which eliminates a great deal.

ARP. The command that shows which MAC address an IP address resolved to. This is how you catch a duplicate address: one address resolving to two different MAC addresses at different moments.

The switch's own counters. Port errors, utilisation and the MAC address table. A device that is not in the MAC address table is not reaching the switch, whatever its front panel says.

Packet capture. The last resort and the definitive one. If the stream is not in the capture at the receiver, the network is not delivering it and the AV application is innocent. If it is in the capture and the application says nothing is arriving, the network is innocent. Nothing else settles that argument.

The faults that repeat

AV network faults are not evenly distributed. A handful account for most of what you will see, and recognising them by their signature is most of the speed in experienced troubleshooting.

How to do it

1. Write down the symptom in checkable terms, including what still works.
2. Decide which layer it is at, and prove the layer below before going up.
3. Halve the system and test; repeat until the fault is in one place.
4. Change one thing at a time and record each change.
5. Put back anything that did not help, before trying the next thing.
6. When it works, write down what it was. The same fault will come back on another job.

What goes wrong

Practice it

On a bench system, with somebody else introducing the fault.

Security, Records and Handover

AV equipment is usually the least secure hardware on a building network, and after you configure it, it is yours. The records you leave are what decide whether the next person can work on it at all.

Security on an AV network

Start with the honest position: a great deal of AV gear ships with a default password, runs old firmware, speaks unencrypted protocols, and has no way to be patched quickly. You cannot fix that. What you can do is stop it being reachable from places it has no business being reachable from, and stop the easy things being easy.

The four measures that do most of the work. Change every default password and record them somewhere the organisation controls rather than in your phone. Put AV on its own VLAN so a compromised AV device cannot see the building's data. Do not give AV devices a route to the internet unless something specifically needs one, and if it does, limit it to that device. Manage switches over a separate management VLAN, by a secure protocol, with the insecure ones turned off.

Beyond that, two things are worth understanding rather than implementing. A firewall controls what crosses between networks and is where an AV VLAN meets the building — the conversation to have with IT rather than a setting you change. And an access control list on a switch does a smaller version of the same thing, which is occasionally the right tool when a specific device must only talk to one other.

The threat that actually materialises on AV networks is rarely an attacker. It is a device with a stuck broadcast, or a laptop plugged into the wrong port flooding the segment, or a well-meaning engineer running a discovery scan across a live show network. Segmentation protects you from all three, which is the real argument for it.

Working with the IT department

On most jobs the network is not yours. It belongs to an IT department whose job is stability and security, who have been burned by equipment they did not choose, and who will say no to anything vague. Getting a yes is a skill and it is mostly about arriving prepared.

Ask for specific things, in their language, with numbers. Not "we need multicast" but "we need IGMP snooping enabled on VLAN 40 with a querier, for eleven devices sending a total of 240 megabits of multicast between these two switches". Not "we need QoS" but "we need these DSCP markings trusted on these ports". A request in that form can be assessed, and an IT department that can assess a request can approve it.

Bring the addressing plan, the bandwidth arithmetic and the device list to the first meeting. Say what you will do if they say no. Offer a separate AV network as the alternative, because sometimes it is the right answer and offering it shows you understand their position.

And accept their constraints where they are real. A hospital or a bank has rules you are not going to argue away. The design that works inside the constraints is the design.

Documentation and handover

The network you built will be worked on by somebody who was not there, probably in a hurry, probably at night. What you write down is the entire difference between a twenty-minute fix and a replacement system.

Document the things that cannot be discovered by looking. Anybody can trace a cable; nobody can guess why storm control is set to the number you set it to. So the record has six parts, and a page each is plenty.

Export the switch configurations as files and store them with the documents. A configuration export is the most complete record there is, and it turns a dead switch from a rebuild into a ten-minute swap.

Then hand it over in person if you possibly can, walk somebody through the diagram, and tell them the one thing most likely to go wrong. Every system has one.

How to do it

1. Change and record every default password.
2. Confirm AV is on its own VLAN and has no route it does not need.
3. Export every switch configuration to a file.
4. Draw the diagram, write the addressing plan, and list the VLANs.
5. Write the changed-from-default list and the baseline figures.
6. Hand the pack to the client and to whoever maintains the system, and keep a copy.
7. Walk one person through it and tell them what is most likely to fail.

What goes wrong

Practice it

On a system you have already built or worked on.

AV Networking as a Converged Skill

The trade is short of people who can do both halves of this. That shortage is the opportunity, and it is unusually durable, because the two halves are learned in different places and few people bother to collect both.

The AV/IT skill the industry needs.

Why this skill is in demand

There are a great many competent network engineers and a great many competent AV technicians. The number of people who can walk into a room with a Dante system that is clicking, work out that it is a clock election rather than bandwidth, log into the switch and fix it, and then explain the change to an IT department in their own language — that number is small, and every integrator and venue is trying to hire it.

It is a converged skill in the literal sense: neither trade teaches it. IT training does not cover PTP or why 260 seconds of multicast matters to a show. AV training has historically stopped at "plug it into the switch". The gap is where the work is.

It also travels. A network is a network in a stadium, a studio, a corporate campus or a cruise ship, and the things in this book are the same in all of them. Few AV specialisms are that portable.

Getting better at it on purpose

Three things, in order of how much they pay back.

Get your hands on a managed switch of your own. A second-hand eight-port managed switch costs less than a microphone and you can break it as often as you like. Everything in this book can be practised on one, and practising on a bench is the only way to be fast on a show.

Learn one vendor's command line properly rather than three web interfaces badly. The concepts transfer completely; only the syntax changes. Somebody who can configure a VLAN and a trunk from the command line on one platform can read any other platform's documentation and get there.

Then go and formalise it. CompTIA Network+ is the common first certification and it maps closely onto the general networking half of this book. The manufacturer programmes — Dante certification, the AV-over-IP vendors' own training — cover the AV half. Having both on paper is what gets you into rooms, and the combination is rarer than either.

Read the standard text, too. The book cited throughout this one — Networked AV Systems, published with InfoComm International, which is now AVIXA — is the industry reference and goes considerably further than a course of this length can, particularly on subnetting arithmetic, streaming system design and the commercial side.

Where it leads

Directly onward: AV-over-IP and network design, where you plan networks rather than configure them; IP video engineering, where the video half gets serious; and the systems-integration roles where somebody has to own the whole converged estate.

Sideways: broadcast engineering, which has been doing this longer than anyone and pays accordingly; and the IT side of large venues, where AV networking is increasingly just part of the network team's remit.

The honest encouragement is that this is learnable and the barrier is mostly nerve. A switch is not mysterious. It has a few dozen settings that matter, this book covers most of them, and the rest is reading the manual for the one in front of you.

How to do it

1. Buy or borrow a managed switch and configure it from scratch three times.
2. Build a VLAN, a trunk, IGMP snooping and a QoS policy on it without notes.
3. Do the bandwidth arithmetic for a real system you have worked on.
4. Write the six-part record for a system that has none.
5. Sit down with whoever runs the network at your venue and ask them to show you their configuration.
6. Book one certification and give yourself a date.

What goes wrong

Practice it

An honest audit of where you are.

Glossary

More terms — the switch and the VLAN

The eight terms above are the essentials. These are the rest, grouped the way the book uses them. They are split into four short tables rather than one long one so that no table breaks over a page.

More terms — addresses and discovery

More terms — multicast, priority and clock

More terms — the settings that bite

Competencies this book closes

AV Networking Intermediate is mapped to the AV_NET_INT cluster in the Reboot Gap Library. It builds on Networking Basics in the Networking & IT track, supporting AV-over-IP, Dante, and IP video.

Assessment

Hands-on & sign-off

AV networking is learned configuring real networks. The learner understands switching, VLANs, and AV traffic, configures a network for AV, and connects/troubleshoots AV devices. A mentor signs off when the learner can configure and manage a network for AV.

Where to Go Next

Everything downstream of here assumes this chapter's skills. AV-over-IP and network design takes the planning side — multiple subnets, routed multicast, redundancy and the designs that carry hundreds of streams. IP video engineering takes the video side into compression, ST 2110 and broadcast practice. Both are considerably easier if you can already configure the switch in front of you.

The first ninety days

Get a managed switch of your own and configure it until the procedure in the "Configuring a Switch for AV" chapter takes you twenty minutes instead of an hour. Do the bandwidth arithmetic for every system you work on, even when nobody asks, because it takes two minutes and it is the thing most often skipped. And on the next job, write the six-part record even if it is not in your scope — it is the fastest way to be the person they call next time.

Where this leads

AV network engineer, systems integration, broadcast engineering, and the venue roles where AV and IT have stopped being separate departments. The trade is short of this and shows no sign of catching up.

What the gear actually looks like

Every device named in this book, in one place. Recognition is the whole point of Chapter 4 — use this page to attach a picture to each name before you meet it on a dock at six in the morning.

Audio

Dynamic microphone

Condenser microphone

Wireless handheld

Radio rack

Mixing console

Stage box / snake

Power amplifier

Main loudspeaker

Subwoofer

Stage monitor wedge

In-ear monitors

DI box

Video

Camera

Video switcher

Scaler / processor

Projector

LED wall panel

Confidence monitor

Lighting & control

Conventional fixture

Moving-head fixture

Lighting console

Dimmer

Show controller

Media server

Networking & infrastructure

Network switch

Patch bay

Equipment rack

Power distribution

Cable tester

A properly coiled cable

Connectors, close up

XLR

TRS / TS ¼-inch

SDI on BNC

HDMI

Cat5e / Cat6 (RJ45)

Fiber optic

DMX (5-pin XLR)

AC power — not signal