xcc.es
[READ-ONLY] Mirror of https://github.com/mrgnw/spatial-maker.
6.1 kB
207 lines
1# spatial-maker
2
3Convert 2D images and videos to stereoscopic 3D spatial content for Apple Vision Pro using AI depth estimation.
4
5[](https://crates.io/crates/spatial-maker)
6[](https://docs.rs/spatial-maker)
7
8## Features
9
10- **Fast depth estimation** — CoreML on Apple Silicon (128ms/frame on M4 Pro) with ONNX fallback
11- **High-quality stereo generation** — Depth-Image Based Rendering (DIBR) with hole filling
12- **Photo & video support** — Process single images or full videos with progress callbacks
13- **Multi-format input** — JPEG, PNG, AVIF, JPEG XL, HEIC via native decoders or ffmpeg
14- **MV-HEVC output** — Side-by-side, top-and-bottom, or separate stereo pairs with optional MV-HEVC packaging
15
16## 🚧 massively under construction 🧱
17
18## Quick Start
19
20### Install the CLI
21
22```sh
23cargo install spatial-maker
24```
25
26for the latest code (untested)
27
28```sh
29cargo install --git https://github.com/mrgnw/spatial-maker --force
30```
31
32
33
34### Convert a Video
35
36```bash
37INPUT_VIDEO=~/Movies/my-video.mp4
38MODEL=b # s (small), b (base), or l (large)
39
40spatial-maker video "$INPUT_VIDEO" -o "$OUTPUT_VIDEO" --model "$MODEL"
41```
42
43### Convert a Photo
44
45```bash
46INPUT_PHOTO=~/Pictures/photo.jpg
47MODEL=b
48MAX_DISPARITY=30 # Higher = more 3D depth
49
50spatial-maker photo "$INPUT_PHOTO"
51 --model "$MODEL" \
52 --max-disparity "$MAX_DISPARITY"
53```
54
55### Use as a Library
56
57Add to your `Cargo.toml`:
58
59```toml
60[dependencies]
61spatial-maker = "0.1"
62```
63
64Process a photo:
65
66```rust
67use spatial_maker::{process_photo, SpatialConfig, OutputOptions};
68use std::path::Path;
69
70#[tokio::main]
71async fn main() -> Result<(), Box<dyn std::error::Error>> {
72 let input_photo = Path::new("input.jpg");
73 let output_photo = Path::new("output_sbs.jpg");
74
75 let config = SpatialConfig::default(); // Uses Small model (48MB, auto-downloaded)
76 let output_opts = OutputOptions::default(); // Side-by-side JPEG
77
78 process_photo(input_photo, output_photo, config, output_opts).await?;
79
80 Ok(())
81}
82```
83
84Process a video:
85
86```rust
87use spatial_maker::{process_video, SpatialConfig, VideoProgress};
88use std::path::Path;
89
90#[tokio::main]
91async fn main() -> Result<(), Box<dyn std::error::Error>> {
92 let input_video = Path::new("input.mp4");
93 let output_video = Path::new("output_sbs.mp4");
94 let config = SpatialConfig::default();
95
96 process_video(
97 input_video,
98 output_video,
99 config,
100 Some(Box::new(|progress: VideoProgress| {
101 println!("{}: {:.1}%", progress.stage, progress.percent);
102 })),
103 ).await?;
104
105 Ok(())
106}
107```
108
109## Model Sizes
110
111Models are auto-downloaded from [HuggingFace](https://huggingface.co/mrgnw/depth-anything-v2-coreml) on first use:
112
113| Size | Download | Quality | Speed (M4 Pro) | License |
114|------|----------|---------|----------------|---------|
115| **Small** (default) | 48 MB | Good | ~85ms/frame | Apache-2.0 ✅ |
116| **Base** | 186 MB | Better | ~128ms/frame | CC-BY-NC-4.0 ⚠️ |
117| **Large** | 638 MB | Best | ~190ms/frame | CC-BY-NC-4.0 ⚠️ |
118
119⚠️ Base and Large models are **non-commercial only** (CC-BY-NC-4.0). Small is Apache-2.0 (commercial OK).
120
121Select a model (CLI):
122
123```bash
124MODEL=b # s (small, 48MB), b (base, 186MB), l (large, 638MB)
125INPUT=~/Movies/video.mp4
126OUTPUT=~/Desktop/spatial.mp4
127
128spatial-maker video "$INPUT" -o "$OUTPUT" --model "$MODEL"
129```
130
131Select a model (library):
132
133```rust
134let model_size = "b".to_string(); // "s", "b", or "l"
135let max_disparity = 30;
136
137let config = SpatialConfig {
138 encoder_size: model_size,
139 max_disparity,
140 ..Default::default()
141};
142```
143
144## Feature Flags
145
146```toml
147[dependencies]
148spatial-maker = { version = "0.1", features = ["onnx"] }
149```
150
151| Feature | Default | Description |
152|---------|---------|-------------|
153| `coreml` | ✅ | CoreML via Swift FFI (macOS only, ~3x faster than ONNX CPU) |
154| `onnx` | ❌ | ONNX Runtime fallback (cross-platform) |
155| `avif` | ❌ | Native AVIF decoder (requires system libdav1d) |
156| `jxl` | ❌ | Native JPEG XL decoder (pure Rust via jxl-oxide) |
157| `heic` | ❌ | Native HEIC decoder (requires system libheif) |
158
159Formats not enabled via feature flags fall back to ffmpeg conversion (slower but works for everything).
160
161## How It Works
162
1631. **Depth Estimation**: [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) via CoreML (default) or ONNX
1642. **Preprocessing**: Resize to 518×518, normalize with ImageNet stats, NCHW tensor format
1653. **Stereo Generation**: DIBR shifts pixels based on depth; expanding-ring hole filling for disocclusions
1664. **Video Pipeline**: `ffmpeg` frame extraction → depth+stereo (parallelized) → `ffmpeg` encoding with model caching
167
168## API Overview
169
170```rust
171// High-level API (most common)
172pub async fn process_photo(input, output, config, output_options) -> SpatialResult<()>
173pub async fn process_video(input, output, config, progress_cb) -> SpatialResult<()>
174
175// Low-level API (custom pipelines)
176pub struct CoreMLDepthEstimator; // macOS CoreML backend
177pub struct OnnxDepthEstimator; // ONNX Runtime backend
178pub fn generate_stereo_pair(image, depth, max_disparity) -> SpatialResult<(DynamicImage, DynamicImage)>
179pub fn create_sbs_image(left, right) -> DynamicImage
180```
181
182See [docs.rs/spatial-maker](https://docs.rs/spatial-maker) for full API documentation.
183
184## Requirements
185
186### For CoreML (default, macOS only):
187- macOS 13.0+ (Ventura or later)
188- Apple Silicon recommended (falls back to CPU on Intel)
189- Xcode Command Line Tools (for Swift compilation during build)
190
191### For ONNX (optional, cross-platform):
192- Any OS (Linux, Windows, macOS)
193- Binaries auto-downloaded via `ort` crate
194
195### For Video:
196- `ffmpeg` in PATH — [https://ffmpeg.org](https://ffmpeg.org)
197- `spatial` CLI for MV-HEVC output — [https://blog.mikeswanson.com/spatial](https://blog.mikeswanson.com/spatial)
198
199```bash
200brew install ffmpeg spatial
201```
202
203## License
204
205MIT
206
207Models: Small (Apache-2.0), Base/Large (CC-BY-NC-4.0). See [HuggingFace repo](https://huggingface.co/mrgnw/depth-anything-v2-coreml) for details.