steroscopic-hardware is an open-source project for real-time stereoscopic depth mapping using Zedboards and a Go-based webserver. It streams synchronized video feeds from two Zedboards, computes a depth map in hardware, and provides a WebUI for visualization and control.
- Real-time stereo video streaming from dual Zedboards
- Hardware-accelerated depth map calculation
- Web-based user interface for live viewing and control
- Prebuilt binaries for easy deployment
- Nix-based reproducible development environment
main.go– Entry point for the Go webservercmd/– Command-line and web server componentspkg/– Core Go packages (camera, despair, logger, lzma, web, etc.)static/– Static assets for the WebUI (JS, CSS, icons)assets/– Images and UI previewsimage_capture/,image_receive/– C code for image acquisition/processingtestdata/– Example images for testing
- Hardware: 2x Zedboards (or compatible FPGA boards)
- Software:
- Zedboards capture synchronized video streams and send data to the Go webserver.
- Go Webserver receives, processes, and streams the feeds, computes the depth map, and serves the WebUI.
- WebUI displays live video, depth map, and provides controls for users.
Download the latest release here
Included in the repository is a prebuilt webserver binary. (See the release section)
To run it, simply download the respective binary for your platform and run it.
To develop the webserver, you need to have the following installed:
Then, run the following commands (from the root of the repository):
# Install dependencies
go mod tidy
# Run Code Generation Step
go generate ./...
# Run the webserver
go run main.goThis will start the webserver on port 8080.
To develop using the development environment, you need to have nix installed.
From the root of the repository, run the following commands:
direnv allowThis will allow direnv to automatically load the environment variables and development dependencies.
Contributions, bug reports, and feature requests are welcome! Please open an issue or submit a pull request.
For questions or support, open an issue or contact the maintainer via GitHub.
The Stereoscopic Hardware Project is an ambitious open-source initiative aimed at real-time depth mapping using stereo vision. Currently, the project has established a solid foundation with a well-structured architecture that integrates hardware camera systems with software processing capabilities. The system architecture follows a modular design that separates concerns across different packages, making the codebase maintainable and extensible.
The project is built around a Go-based web server that acts as the central processing hub. This server interfaces with stereoscopic cameras (either physical hardware connected via serial ports or simulated through static images), processes the resulting image pairs, and generates depth maps that visually represent distance information. The architecture is divided into several key components:
-
Camera System: The
pkg/camerapackage implements a robust camera management system with multiple camera types:SerialCamera: Interfaces with physical cameras connected via serial portsStaticCamera: Loads images from files for development and testingOutputCamera: Processes stereo image pairs to generate depth maps
-
Depth Mapping Algorithm: The
pkg/despairpackage contains the implementation of the Sum of Absolute Differences (SAD) algorithm, which is the core of the stereoscopic processing. This algorithm:- Takes left and right grayscale images
- Compares blocks of pixels between the images
- Calculates horizontal displacement (disparity)
- Generates a grayscale disparity map where pixel brightness represents depth
-
Web Interface: The project features a modern web UI built with:
- HTMX for dynamic content updates without full page reloads
- AlpineJS for reactive components and UI state management
- Tailwind CSS for styling
- SVG templates for icons and visual elements
-
API Layer: The
cmd/handlerspackage provides a comprehensive API for camera control, configuration, and image streaming.
The project currently offers:
-
Dual Camera Visualization: The web interface displays both left and right camera feeds side-by-side, allowing for real-time monitoring of the stereo inputs.
-
Depth Map Generation: The system processes the stereo image pairs in real-time to produce a depth map visualization.
-
Algorithm Parameter Tuning: Users can adjust key parameters like block size (3-31) and maximum disparity (16-256) through intuitive slider controls in the web UI.
-
Camera Configuration: The interface allows configuration of camera settings including:
- Serial port selection with auto-detection
- Baud rate configuration
- Compression settings
-
Static Image Testing: For development without hardware, the system supports uploading static image files that can be used in place of live camera feeds.
-
Logging System: A comprehensive logging system captures application events and errors, with the ability to view logs in the UI.
The codebase demonstrates several advanced technical features:
-
Concurrent Processing: The depth mapping algorithm leverages Go's concurrency model with worker pools for efficient parallel processing of image chunks.
-
Performance Optimizations:
- Direct pixel access for faster image processing
- Chunked processing for parallel computation
- Early termination in comparison loops
- Optimized bounds checking
- Precomputed lookup tables for common conversions
-
Memory Management: The image processing pipeline is designed with memory efficiency in mind, using buffer pooling and reuse where appropriate.
-
Error Handling: The system implements robust error handling throughout, with graceful degradation and meaningful error reporting.
The project employs modern development practices:
-
Reproducible Development: Uses Nix for creating a consistent development environment across different machines.
-
Hot Reloading: Integrates with tools like Air for rapid development with automatic code reloading.
-
Linting and Testing: Employs Golangci-lint for code quality and has a test suite for the core algorithms.
-
Documentation: Each package includes comprehensive documentation through Go's standard doc comments, which are processed with gomarkdoc to generate Markdown documentation.
import "github.com/conneroisu/steroscopic-hardware/cmd"Package cmd implements the application's web server and HTTP API for stereoscopic image processing.
The cmd package serves as the entry point for the application, providing:
- A web server with a UI for controlling the stereoscopic cameras
- API endpoints for camera configuration and image streaming
- Depth map generation from stereo image pairs
- Graceful shutdown handling
The main packages are:
- Server: HTTP server implementation with proper timeouts
- Routes: API endpoint definitions for camera control and streaming
- Components: Templ-based UI components for web interface
- Handlers: HTTP handlers for API endpoints
- func AddRoutes(ctx context.Context, mux *http.ServeMux, logger *logger.Logger, cancel context.CancelFunc) error
- func NewServer(ctx context.Context, logger *logger.Logger, cancel context.CancelFunc) (http.Handler, error)
- func Run(ctx context.Context, onStart func()) error
func AddRoutes
func AddRoutes(ctx context.Context, mux *http.ServeMux, logger *logger.Logger, cancel context.CancelFunc) errorAddRoutes configures all HTTP routes and handlers for the application.
This function registers endpoints for camera control, streaming, and UI components.
func NewServer
func NewServer(ctx context.Context, logger *logger.Logger, cancel context.CancelFunc) (http.Handler, error)NewServer creates a new web-ui server with all necessary routes and handlers configured.
It sets up the HTTP server with routes for camera streaming, configuration, and depth map generation. The server includes logging middleware that captures request information.
Parameters:
- logger: The application logger for recording events and errors
- params: Stereoscopic algorithm parameters (block size, max disparity)
- cancel: CancelFunc to gracefully shut down the application
Returns an http.Handler and any error encountered during setup.
func Run
func Run(ctx context.Context, onStart func()) errorRun is the entry point for the application that starts the HTTP server and manages its lifecycle.
Process:
- Sets up signal handling for graceful shutdown
- Initializes the logger and camera system
- Creates and configures the HTTP server with appropriate timeouts
- Starts the server and monitors for shutdown signals
- Performs graceful shutdown when terminated
Http handlers for the web ui can be found here.
import "github.com/conneroisu/steroscopic-hardware/cmd/handlers"Package handlers contains functions for handling API requests.
This Go package `handlers` is part of a stereoscopic hardware system project that manages HTTP requests for a web UI controlling stereo cameras.
It handles the communication between the web interface and the physical camera hardware.
- `Make()` converts these API functions into standard HTTP handlers, with built-in error handling
- `ErrorHandler()` wraps API functions to provide formatted HTML error responses (using color-coded success/failure messages)
-
**Camera Configuration (`ConfigureCamera`):** - Processes form data for camera setup (port, baud rate, compression) - Configures either left or right camera streams based on parameters - Creates new output streams after successful configuration - Includes validation and error handling for all input parameters
-
**Parameters Management (`ParametersHandler`):** - Handles changes to disparity map generator parameters - Processes form data for block size and maximum disparity values - Uses mutex locking to ensure thread safety when updating shared parameters - Logs parameter changes for debugging
-
**Port Discovery (`GetPorts`):** - Enumerates and returns available serial ports as HTML options - Implements retry logic (up to 10 attempts) if ports aren't initially found - Formats port information for direct use in form select elements
-
**Image Streaming (`StreamHandlerFn`):** - Sets up MJPEG streaming with multipart boundaries - Manages client registration and connection lifecycle - Implements performance optimizations: - Buffer pooling to minimize memory allocation - JPEG quality control and compression - Frame rate limiting (10 FPS) - Connection timeouts (30 minutes) - Efficient image encoding with reusable buffers
- `MorphableHandler()` supports HTMX integration by detecting the presence of HX-Request headers
- Thread safety with mutex locks for parameter updates
- Memory efficiency through object pooling (JPEG options)
- Graceful error handling with formatted responses
- Efficient image streaming with buffer reuse
- Robust port detection with retry mechanisms
- Context-aware logging throughout the system
This package serves as the interface layer between the web UI and the underlying stereoscopic hardware, providing both configuration management and real-time image streaming capabilities.
- func HandleLeftStream(w http.ResponseWriter, r *http.Request) error
- func HandleOutputStream(w http.ResponseWriter, r *http.Request) error
- func HandleRightStream(w http.ResponseWriter, r *http.Request) error
- func Make(fn APIFn) http.HandlerFunc
- func MorphableHandler(wrapper func(templ.Component) templ.Component, morph templ.Component) http.HandlerFunc
func HandleLeftStream
func HandleLeftStream(w http.ResponseWriter, r *http.Request) errorHandleLeftStream returns a handler for streaming the left camera.
func HandleOutputStream
func HandleOutputStream(w http.ResponseWriter, r *http.Request) errorHandleOutputStream returns a handler for streaming the output camera.
func HandleRightStream
func HandleRightStream(w http.ResponseWriter, r *http.Request) errorHandleRightStream returns a handler for streaming the right camera.
func Make
func Make(fn APIFn) http.HandlerFuncMake returns a function that can be used as an http.HandlerFunc.
func MorphableHandler
func MorphableHandler(wrapper func(templ.Component) templ.Component, morph templ.Component) http.HandlerFuncMorphableHandler returns a handler that checks for the presence of the hx-trigger header and serves either the full or morphed view.
APIFn is a function that handles an API request.
func ConfigureCamera
ConfigureCamera handles client requests to configure camera parameters.
func ConfigureMiddleware
func ConfigureMiddleware(apiFn APIFn) APIFnConfigureMiddleware parses camera configuration from form data.
It adds the configuration to the request context.
This middleware is required for the ConfigureCamera handler.
func ErrorHandler
func ErrorHandler(fn APIFn) APIFnErrorHandler returns a handler that returns an error response.
func GetPorts
func GetPorts(logger *logger.Logger) APIFnGetPorts handles client requests to configure the camera.
func HandleCameraStream
HandleCameraStream is a generic handler for streaming camera images.
func ParametersHandler
func ParametersHandler() APIFnParametersHandler handles client requests to update disparity algorithm parameters.
import "github.com/conneroisu/steroscopic-hardware/pkg/despair"Package despair provides a Go implementation of a stereoscopic depth mapping algorithm, designed for efficient generation of depth/disparity maps from stereo image pairs.
The package implements the Sum of Absolute Differences (SAD) algorithm, a common technique in stereoscopic vision that:
- Takes left and right grayscale images from slightly different viewpoints
- Compares blocks of pixels to find matching points between images
- Calculates disparity (horizontal displacement) between matching points
- Generates a grayscale disparity map where pixel brightness represents depth
InputChunk: Represents a portion of the image pair to process
OutputChunk: Contains processed disparity data for a specific region
Parameters: Configuration settings for the algorithm including:
`BlockSize`: Size of pixel blocks for comparison
`MaxDisparity`: Maximum pixel displacement to check
-
`SetupConcurrentSAD`: Creates a pipeline with configurable worker count, returning input/output channels
-
`RunSad`: Convenience function that orchestrates the entire process: - Divides images into chunks - Distributes processing across workers - Assembles final disparity map
-
`AssembleDisparityMap`: Combines processed chunks into a complete disparity map
-
`sumAbsoluteDifferences`: Low-level function that calculates block matching scores
The package includes efficient image handling utilities:
- PNG Loading/Saving: Optimized functions for loading and saving grayscale PNG images
- Error Handling: Both standard error-returning functions and "Must" variants that panic on failure
- Concurrent Processing: Utilizes Go's concurrency with multiple worker goroutines
- Chunked Processing: Splits images into smaller regions for parallel processing
- Direct Pixel Access: Works with underlying pixel arrays rather than the higher-level interface
- Early Termination: Breaks comparison loops when perfect matches are found
- Optimized Bounds Checking: Reduces redundant checks in inner loops
- Precomputed Lookup Tables: Uses LUTs for common conversions
Example:
// Load stereo image pair
left := despair.MustLoadPNG("left.png")
right := despair.MustLoadPNG("right.png")
// Generate disparity map with block size 9 and max disparity 64
disparityMap := despair.RunSad(left, right, 9, 64)
// Save the result
despair.MustSavePNG("depth_map.png", disparityMap)- func AssembleDisparityMap(outputChan <-chan OutputChunk, dimensions image.Rectangle, chunks int) *image.Gray
- func RunSad(left, right *image.Gray, blockSize, maxDisparity int) *image.Gray
- func SetDefaultParams(params Parameters)
- func SetupConcurrentSAD(numWorkers int) (chan<- InputChunk, <-chan OutputChunk)
- func SumAbsoluteDifferences(left, right *image.Gray, leftX, leftY, rightX, rightY, blockSize int) int
func AssembleDisparityMap
func AssembleDisparityMap(outputChan <-chan OutputChunk, dimensions image.Rectangle, chunks int) *image.GrayAssembleDisparityMap assembles the disparity map from output chunks.
func LoadPNG
LoadPNG loads a PNG image and converts it to grayscale with optimizations.
func MustLoadPNG
MustLoadPNG loads a PNG image and converts it to grayscale with optimizations and panics if an error occurs.
func MustSavePNG
func RunSad
func RunSad(left, right *image.Gray, blockSize, maxDisparity int) *image.GrayRunSad is a convenience function that sets up the pipeline, feeds the images, and assembles the disparity map.
This is not used in the web UI, but is useful for testing.
func SavePNG
func SetDefaultParams
func SetDefaultParams(params Parameters)SetDefaultParams sets the default stereoscopic algorithm parameters.
func SetupConcurrentSAD
func SetupConcurrentSAD(numWorkers int) (chan<- InputChunk, <-chan OutputChunk)SetupConcurrentSAD sets up a concurrent SAD processing pipeline.
It returns an input channel to feed image chunks into and an output channel to receive results from.
If the input channel is closed, the processing pipeline will stop.
func SumAbsoluteDifferences(left, right *image.Gray, leftX, leftY, rightX, rightY, blockSize int) intSumAbsoluteDifferences calculates SAD directly on image data.
InputChunk represents a portion of the image to process.
Left, Right *image.Gray
Region image.Rectangle
}OutputChunk represents the processed disparity data for a region.
DisparityData []uint8
Region image.Rectangle
}Parameters is a struct that holds the parameters for the stereoscopic image processing.
BlockSize int `json:"blockSize"`
MaxDisparity int `json:"maxDisparity"`
}func DefaultParams
func DefaultParams() *ParametersDefaultParams returns the default stereoscopic algorithm parameters.
Package logger provides a multi faceted logger that
import "github.com/conneroisu/steroscopic-hardware/pkg/logger"Package logger provides a multi faceted logger that can be used to log to the console and a buffer.
It's intended to be used as a default logger for the application.
Allowing for the logging of console messages both to the console and to the browser.
func NewLogWriter
func NewLogWriter(w io.Writer) slog.HandlerNewLogWriter returns a slog.Handler that writes to a buffer.
LogEntry represents a structured log entry.
Level slog.Level
Time time.Time
Message string
Attrs []slog.Attr
}Logger is a slog.Logger that sends logs to a channel and also to the console.
*slog.Logger
// contains filtered or unexported fields
}func NewLogger
func NewLogger() LoggerNewLogger creates a new Logger.
func (Logger) Bytes
func (l Logger) Bytes() []byteBytes returns the buffered log.
Package lzma package implements reading and writing of LZMA format compressed data.
import "github.com/conneroisu/steroscopic-hardware/pkg/lzma"Package lzma package implements reading and writing of LZMA format compressed data.
Reference implementation is LZMA SDK version 4.65 originally developed by Igor Pavlov, available online at:
http://www.7-zip.org/sdk.html
Usage examples. Write compressed data to a buffer:
var b bytes.Buffer
w := lzma.NewWriter(&b)
w.Write([]byte("hello, world\n"))
w.Close()
read that data back:
r := lzma.NewReader(&b)
io.Copy(os.Stdout, r)
r.Close()
If the data is bigger than you'd like to hold into memory, use pipes. Write compressed data to an io.PipeWriter:
pr, pw := io.Pipe()
go func() {
defer pw.Close()
w := lzma.NewWriter(pw)
defer w.Close()
// the bytes.Buffer would be an io.Reader used to read uncompressed data from
io.Copy(w, bytes.NewBuffer([]byte("hello, world\n")))
}()
and read it back:
defer pr.Close()
r := lzma.NewReader(pr)
defer r.Close()
// the os.Stdout would be an io.Writer used to write uncompressed data to
io.Copy(os.Stdout, r)
--------------------------- | Offset | Size | Description | |--------|------|-------------| | 0 | 1 | Special LZMA properties (lc,lp, pb in encoded form) | | 1 | 4 | Dictionary size (little endian) | | 5 | 8 | Uncompressed size (little endian). Size -1 stands for unknown size |
- Constants
- func NewReader(r io.Reader) io.ReadCloser
- func NewWriter(w io.Writer) (io.WriteCloser, error)
- func NewWriterLevel(w io.Writer, level int) (io.WriteCloser, error)
- func NewWriterSize(w io.Writer, size int64) (io.WriteCloser, error)
- func NewWriterSizeLevel(w io.Writer, size int64, level int) (io.WriteCloser, error)
const (
// BestSpeed is the fastest compression level.
BestSpeed = 1
// BestCompression is the compression level that gives the best compression ratio.
BestCompression = 9
// DefaultCompression is the default compression level.
DefaultCompression = 5
)func NewReader
func NewReader(r io.Reader) io.ReadCloserNewReader returns a new io.ReadCloser that can be used to read the uncompressed version of `r`
It is the caller's responsibility to call Close on the io.ReadCloser when finished reading.
func NewWriter
func NewWriter(w io.Writer) (io.WriteCloser, error)NewWriter creates a new Writer that compresses data to the given Writer using the default compression level.
Same as NewWriterSizeLevel(w, -1, DefaultCompression).
func NewWriterLevel
func NewWriterLevel(w io.Writer, level int) (io.WriteCloser, error)NewWriterLevel creates a new Writer that compresses data to the given Writer using the given level.
Level is any integer value between lzma.BestSpeed and lzma.BestCompression.
Same as lzma.NewWriterSizeLevel(w, -1, level).
func NewWriterSize
func NewWriterSize(w io.Writer, size int64) (io.WriteCloser, error)NewWriterSize creates a new Writer that compresses data to the given Writer using the given size as the uncompressed data size.
If size is unknown, use -1 instead.
Level is any integer value between lzma.BestSpeed and lzma.BestCompression.
Parameter size and the size, lzma.DefaultCompression, (the lzma header) are written to the passed in writer before any compressed data.
If size is -1, last bytes are encoded in a different way to mark the end of the stream. The size of the compressed data will increase by 5 or 6 bytes.
Same as NewWriterSizeLevel(w, size, lzma.DefaultCompression).
func NewWriterSizeLevel
func NewWriterSizeLevel(w io.Writer, size int64, level int) (io.WriteCloser, error)NewWriterSizeLevel writes to the given Writer the compressed version of data written to the returned io.WriteCloser. It is the caller's responsibility to call Close on the io.WriteCloser when done.
Parameter size is the actual size of uncompressed data that's going to be written to io.WriteCloser. If size is unknown, use -1 instead.
Parameter level is any integer value between lzma.BestSpeed and lzma.BestCompression.
Arguments size and level (the lzma header) are written to the writer before any compressed data.
If size is -1, last bytes are encoded in a different way to mark the end of the stream. The size of the compressed data will increase by 5 or 6 bytes.
The reason for which size is an argument is that, unlike gzip which appends the size and the checksum at the end of the stream, lzma stores the size before any compressed data. Thus, lzma can compute the size while reading data from pipe.
An ArgumentValueError reports an error encountered while parsing user provided arguments.
// contains filtered or unexported fields
}func (*ArgumentValueError) Error
func (e *ArgumentValueError) Error() stringHeaderError is returned when the header is corrupt.
// contains filtered or unexported fields
}func (HeaderError) Error
func (e HeaderError) Error() stringNWriteError is returned when the number of bytes returned by Writer.Write() didn't meet expectances.
// contains filtered or unexported fields
}func (*NWriteError) Error
func (e *NWriteError) Error() stringReader is the actual read interface needed by [NewDecoder].
If the passed in io.Reader does not also have ReadByte, the [NewDecoder] will introduce its own buffering.
io.Reader
ReadByte() (c byte, err error)
}StreamError is returned when the stream is corrupt.
// contains filtered or unexported fields
}func (*StreamError) Error
func (e *StreamError) Error() stringWriter is the actual write interface needed by [NewEncoder].
If the passed in io.Writer does not also have WriteByte and Flush, the [NewEncoder] function will wrap it into a bufio.Writer.
io.Writer
Flush() error
WriteByte(c byte) error
}import "github.com/conneroisu/steroscopic-hardware/pkg/web"Package web contains SVG templates and dom targets for the web UI.
SVG templates are used to render SVG icons and text in the web UI. Templates are embedded into the package using the go:embed directive.
var (
// TargetLogContainer is a target for the log container.
// It is used to insert log entries into the DOM.
TargetLogContainer = Target{
ID: "log-container",
Sel: "#log-container",
}
TargetStatusContent = Target{
}
)CircleQuestion is a template for the SVG circle-question icon.
var CircleQuestion = templ.Raw(circleQuestion)CircleX is a template for the SVG circle-x icon.
var CircleX = templ.Raw(circleX)GreenUp is a template for the SVG green-up icon.
var GreenUp = templ.Raw(greenUp)var (
// LivePageTitle is the title of the live page.
LivePageTitle = "Live Camera System"
)RedDown is a template for the SVG red-down icon.
var RedDown = templ.Raw(redDown)RefreshCw is a template for the SVG refresh-cw icon.
var RefreshCw = templ.Raw(refreshCw)SettingsGear is a template for the SVG settings-geat icon.
var SettingsGear = templ.Raw(settingsGear)Target is a struct representing a dom target.
ID string `json:"id"`
Sel string `json:"sel"`
}