WIZ-Cam Data Flow

This page explains how WIZ-Cam captures video, how it gets frames onto the network without holding a whole frame in RAM, how it streams microphone audio at the same time, and how it plays audio on the speaker. It is background for anyone changing the firmware; you do not need it to use the board.

The whole board at a glance:

OV2640 ------------> DMA1 --> capture ring --+
                                             |
ICS-43434 --> I2S3 --> DMA2 --> audio ring --+--> W55MH32 <--> Ethernet
                                             |
MAX98357A <-- I2S3 <-- DMA2 <-- audio ring <-+

The same data moves along three separate paths.

Video, from the camera to the network

OV2640  ->  DMA1  ->  capture ring in RAM  ->  W55MH32  ->  Ethernet

The data lines (D0-D7) carry the bytes. PCLK and HREF go to TIM1, which tells DMA1 when to copy each byte.

Audio, from the microphone to the network

ICS-43434  ->  I2S3  ->  DMA2  ->  audio ring in RAM  ->  W55MH32  ->  Ethernet

Audio, from the network to the speaker

Ethernet  ->  W55MH32  ->  audio ring in RAM  ->  DMA2  ->  I2S3  ->  MAX98357A

The two audio paths share I2S3, so only one of them runs at a time.

Capturing the Camera with a Gated Slave Timer

The problem

The OV2640 outputs one byte per pixel clock (PCLK) on eight data lines, D0-D7. A byte is only valid while HREF is high; between lines and between frames the data lines carry nothing useful. VSYNC marks the start and end of each frame.

Reading those lines in a software loop does not work at useful speeds. Even a tight @micropython.viper loop that polls the GPIO register tops out at a few MHz of sampling, and it must catch every single PCLK edge: one missed edge shifts every following byte and corrupts the JPEG. The W55MH32 has no dedicated camera interface (DCMI), so the capture has to be built from a general-purpose timer and DMA.

The solution

TIM1 is set up so that the hardware itself decides when a byte is valid:

PCLK drives the counter. PCLK is wired to PA12, which is TIM1’s external trigger input (ETR). In external clock mode 2, every rising PCLK edge advances the TIM1 counter.
HREF gates the counter. HREF is wired to PA8, TIM1 channel 1. In gated slave mode, with channel 1 as the trigger input, the counter only runs while HREF is high. While HREF is low, PCLK edges are ignored.
Every counted edge is a DMA request. The auto-reload value is 0, so every counted edge produces an update event. The capture/compare DMA request is routed to the update event, so each valid pixel clock asks DMA1 for one transfer.
The DMA copies one byte per request. The DMA channel reads the low byte of the GPIOA input register (D0-D7 are PA0-PA7) and writes it into a circular buffer in RAM, the capture ring.

The result is that bytes are copied only on valid pixel clocks, with the line gating done entirely in hardware. The CPU does not touch individual bytes at all; it only watches how far the DMA has got.

Frame synchronisation

VSYNC (PC5) is the only signal the CPU follows directly. Before each frame, the firmware waits for the gap between frames, re-arms the DMA at the start of the ring, and enables the timer’s DMA request before the new frame begins. Enabling it early matters: the JPEG stream starts right at the VSYNC edge, so enabling the request after the edge can miss the first byte (FF of the FF D8 start marker). Because the HREF gate keeps the timer stopped until real data arrives, enabling early captures nothing extra.

When VSYNC shows the frame has ended, the firmware stops the DMA and knows exactly how many bytes the frame used.

Note

PA12 is also the USB D+ pin. With the USB peripheral clocked, the pixel clock on that pin looks like bus activity to the USB core and triggers interrupts that take CPU time away from streaming. WIZ-Cam therefore keeps the USB peripheral clock off.

Section-to-Section Sending

The problem

A JPEG frame from the OV2640 can be tens of kilobytes; at 1280x960 it can be larger than all the free RAM left after the network stack and MicroPython. Capturing a whole frame and then sending it would also leave the network idle during capture and the camera idle during sending, which halves the frame rate.

The solution

WIZ-Cam never holds a whole frame. The capture ring is smaller than a frame, and the DMA keeps writing into it in a circle while the CPU sends the part that has already arrived. Capture and sending overlap, so the frame rate is set by the sensor, not by “capture time + send time”.

HTTP mode. The ring is split into four equal sections. As soon as the DMA has filled a section, the firmware writes it to the TCP connection while the DMA fills the next one. When the DMA wraps around to the first section, the CPU has already sent it. At the end of the frame (VSYNC), the remaining partial section is sent and the multipart boundary closes the image. Because each image in a multipart/x-mixed-replace stream is delimited by a boundary rather than by a length, the frame size does not need to be known in advance.

RTSP mode. The same idea is applied at a finer grain: instead of whole sections, the firmware sends one RTP packet as soon as a packet’s worth of data is waiting. This keeps the sender very close to the DMA, so RTSP mode needs a much smaller ring (8.4 KB compared with 16 KB for HTTP). Before each packet is sent, the firmware checks that the DMA has not already overwritten the bytes being copied. If it has, the frame is closed early instead of sending corrupted data.

Keeping up with the DMA

The scheme only works if the CPU sends data faster than the camera produces it, and never stalls for long. A few rules follow from that:

Garbage collection runs right after a frame starts. The ring is almost empty at that moment, so this is where a collection costs the least. Running it between frames instead can make the loop miss the next VSYNC and skip a whole frame.
No printing while a frame is arriving. A print over the serial console takes milliseconds. That is why debug output is only produced when DEBUG = True, and in RTSP mode only as periodic summaries between frames.
Settings are applied between frames. The UDP command server and the RTSP connection are checked between frames (every 8 frames in RTSP mode), so a setting change never interrupts a frame in progress.

Streaming Audio Alongside Video

Capture

The ICS-43434 microphone is read through I2S3 (PB3 SCK, PA15 WS, PB5 SD) at 8 kHz. Like the camera, it is captured by DMA: DMA2 writes the I2S data continuously into its own circular audio ring. The microphone delivers 24-bit samples; WIZ-Cam keeps the top 16 bits of each.

Because the DMA keeps capturing on its own, the CPU does not have to read the microphone at exact times. It only has to come back to the audio ring often enough that the ring does not fill up, which leaves plenty of slack.

Why not I2S.readinto() and I2S.write()? In blocking mode, these calls wait until the driver’s internal buffer has room or data. WIZ-Cam shares one CPU loop with the camera, so it can’t afford to wait there, and switching to the speaker with I2S.write() was seen to hang. WIZ-Cam therefore uses machine.I2S only to set up the pins and the I2S clock and format, with ibuf = 1 because the driver’s own buffer is never used. The DMA channels are then programmed directly to move data between I2S3 and WIZ-Cam’s own ring.

RTP payloads

In RTSP mode, video and audio are two RTP streams described by the SDP that the board returns to DESCRIBE:

Track	Payload type	Clock	Format
Video	26 (JPEG)	90 kHz	RTP/JPEG as defined in RFC 2435
Audio	96 (dynamic)	8 kHz	L16: 16-bit linear PCM, mono, big-endian

Video. RFC 2435 does not send the JPEG file header. Each packet carries the frame’s type, size and a fragment offset, and the first packet of each frame also carries the quantization tables. The player rebuilds a complete JPEG from these. WIZ-Cam reads the tables and image size from the header the OV2640 produces, then sends only the image data. Packets are sized to fit a single Ethernet frame, and the last packet of each frame has the RTP marker bit set.

Audio. L16 is uncompressed 16-bit PCM in network (big-endian) byte order, so each sample’s bytes are swapped from the microphone’s little-endian order when the packet is built. The RTP timestamp advances by 160 per packet, the number of samples it holds.

Keeping video and audio in sync

An RTP timestamp has an arbitrary starting point and its own clock rate, so a player can’t line up two streams from timestamps alone. Every 5 seconds the board sends an RTCP sender report for each stream, on the RTP port + 1. Both reports use the same clock reading, which ties the two timelines together.

The audio report carries the real audio timeline, which follows the I2S clock. That clock runs at about 7997.6 Hz rather than exactly 8000 Hz, and the repeated reports let the player measure the true rate instead of slowly drifting.

Playing Audio on the Speaker

One I2S block, two directions

The ICS-43434 and the MAX98357A share I2S3, and an I2S block can receive or transmit but not both. The audio command therefore turns the block around: it stops the capture DMA, reconfigures I2S3 as a transmitter, clears the shared audio ring and starts DMA2 channel 2 reading from it. The UDP socket on port 6000 is opened only while the speaker is on. Each new RTSP PLAY switches back to the microphone, so a session that ended during playback can’t leave the next one without a microphone.

The amplifier receives 32-bit stereo frames even though the audio is 16-bit mono. At 8 kHz, 16-bit data would need an I2S clock divider above the 511 maximum, while 32-bit data needs 422, the same divider the microphone uses. Each sample is written to both channels because the amplifier plays the average of left and right.

Because the DMA reads the ring directly, nothing rearranges the samples the way I2S.write() does. The ring is filled in the layout I2S3 sends, with the most significant half of each 32-bit word first.

Receiving audio over UDP

The ring holds 120 ms of audio, in six 20 ms slots. The DMA loops over it at a fixed rate and never waits for the network, so playback starts writing a few slots ahead of the DMA to absorb network jitter.

Each packet’s sequence number decides which slot it goes in, so loss and reordering are handled by position rather than by arrival order:

Missing packets play as silence, not as old audio.
Late packets are dropped, because the DMA has already played their slot. This is also why playback uses UDP: a retransmission would always arrive too late.
Packets that arrive too early are dropped when writing them would overwrite audio that hasn’t played. This absorbs the clock difference between the PC and the board.

One slot of silence is kept ahead of the newest audio, so an underrun is quiet instead of replaying old sound.

Sharing the CPU with video

Reading the socket takes CPU time that the video loop needs while a frame is arriving. So the socket is drained fully between frames, and during a frame the loop takes at most 2 packets, no more often than every 20 ms. Because this runs inside the video loop, the lowest frame rates feed the ring least evenly, which is why UXGA can be choppy.

Keeping the RTSP audio track alive

While the speaker is on there is nothing to capture, but players that sync to the audio clock can freeze the video if audio stops arriving. The board therefore keeps sending 20 ms RTP packets of silence on the audio track, with the same timeline, so the player’s clock keeps running.

If you change this code

Keep w_idx on a slot boundary. w_idx must stay a multiple of the 1280-byte slot. Viper doesn’t check bounds, so otherwise _expand() writes past the ring and corrupts the heap. This has shown up as the Microphone object randomly losing attributes.
Advance the sequence number when dropping an early packet. In _recv_pcm(), a packet dropped for arriving too early must also move the expected sequence number forward. Without this, the amount of queued audio oscillated and about 38% of packets were dropped.
Zero the whole ring before playback. _expand() writes only the sample bytes of each frame, so to_speaker() must zero the ring first. Otherwise, old microphone data ends up in the low bits of the output.