92 lines
8.3 KiB
Markdown
92 lines
8.3 KiB
Markdown
Texture Loading & Streaming
|
||
|
||
Overview
|
||
- Streaming cache: `src/core/texture_cache.{h,cpp}` asynchronously decodes images (stb_image) on a small worker pool (1–4 threads, clamped by hardware concurrency) and uploads them via `ResourceManager` with optional mipmaps. For FilePath keys, a sibling `<stem>.ktx2` (or direct `.ktx2`) is preferred over PNG/JPEG. Descriptors registered up‑front are patched in‑place once the texture becomes resident. Large decodes can be downscaled on workers before upload to cap peak memory.
|
||
- Uploads: `src/core/vk_resource.{h,cpp}` stages pixel data and either submits immediately or registers a Render Graph transfer pass. Mipmaps use `vkutil::generate_mipmaps(...)` and finish in `SHADER_READ_ONLY_OPTIMAL`.
|
||
- Integration points:
|
||
- Materials: layouts use `UPDATE_AFTER_BIND`; descriptors can be rewritten after bind.
|
||
- glTF loader: `src/scene/vk_loader.cpp` builds keys, requests handles, and registers descriptor patches with the cache.
|
||
- Primitives/adhoc: `src/core/asset_manager.cpp` builds materials and registers texture watches.
|
||
- Visibility: `src/render/vk_renderpass_geometry.cpp` and `src/render/vk_renderpass_transparent.cpp` call `TextureCache::markSetUsed(...)` for sets that are actually drawn.
|
||
|
||
Data Flow
|
||
- Request
|
||
- Build a `TextureCache::TextureKey` (FilePath or Bytes), set `srgb` and `mipmapped`.
|
||
- Call `request(key, sampler)` → returns a stable `TextureHandle`, deduplicated by a 64‑bit FNV‑1a hash. For FilePath keys the path plus the sRGB bit are hashed; for Bytes keys the payload hash is XOR’d with a constant when `srgb=true`.
|
||
- Register target descriptors via `watchBinding(handle, set, binding, sampler, fallbackView)`.
|
||
- Visibility‑gated scheduling
|
||
- `pumpLoads(...)` looks for entries in `Unloaded` or `Evicted` state that were seen recently (`now == 0` or `now - lastUsed <= 1`) and starts at most `max_loads_per_pump` decodes per call, while enforcing a byte budget for uploads per frame.
|
||
- Render passes mark used sets each frame with `markSetUsed(...)` (or specific handles via `markUsed(...)`).
|
||
- Decode
|
||
- FilePath: if the path ends with `.ktx2` or a sibling exists, we load via libktx (and transcode to BCn if needed). Otherwise, decode to RGBA8 via stb_image.
|
||
- Bytes: always decode via stb_image (no sibling discovery possible).
|
||
- Admission & Upload
|
||
- Before upload, an expected resident size is computed (exact for KTX2 by summing level byte lengths; estimated for raster by format×area×mip‑factor). A per‑frame byte budget (`max_bytes_per_pump`) throttles uploads.
|
||
- If a GPU texture budget is set, the cache evicts least‑recently‑used textures not used this frame. If it still cannot fit, the decode is deferred or dropped with backoff.
|
||
- Raster: `ResourceManager::create_image(...)` stages a single region, then optionally generates mips on GPU.
|
||
- KTX2: `ResourceManager::create_image_compressed(...)` allocates an image with the file’s `VkFormat` (from libktx) and records one `VkBufferImageCopy` per mip level (no GPU mip gen). Immediate path transitions to `SHADER_READ_ONLY_OPTIMAL`; the RenderGraph path transitions after copy when no mip gen.
|
||
- If the device cannot sample the KTX2 format, the cache falls back to raster decode.
|
||
- After upload: state → `Resident`, descriptors recorded via `watchBinding` are rewritten to the new image view with the chosen sampler and `SHADER_READ_ONLY_OPTIMAL` layout. For Bytes‑backed keys, compressed source bytes are dropped unless `keep_source_bytes` is enabled.
|
||
- Eviction & Reload
|
||
- `evictToBudget(bytes)` rewrites watchers to fallbacks, destroys images, and marks entries `Evicted`. Evicted entries can reload automatically when seen again and a short cooldown has passed (default ~2 frames), avoiding immediate thrash.
|
||
|
||
Runtime UI
|
||
- ImGui → Debug → Textures (see `src/core/vk_engine.cpp`)
|
||
- Shows: device‑local budget/usage (from VMA), texture streaming budget (~35% of device‑local by default), resident MiB, CPU source MiB, counts per state, and a Top‑N table of consumers.
|
||
- Controls: `Loads/Frame`, `Upload Budget (MiB)` (byte‑based throttle), `Keep Source Bytes`, `CPU Source Budget (MiB)`, `Max Upload Dimension` (progressive downscale cap), and `Trim To Budget Now`.
|
||
|
||
Key APIs (src/core/texture_cache.h)
|
||
- `TextureHandle request(const TextureKey&, VkSampler)`
|
||
- `void watchBinding(TextureHandle, VkDescriptorSet, uint32_t binding, VkSampler, VkImageView fallback)`
|
||
- `void unwatchSet(VkDescriptorSet)` — call before destroying descriptor pools/sets
|
||
- `void markSetUsed(VkDescriptorSet, uint32_t frameIndex)` and `void markUsed(TextureHandle, uint32_t frameIndex)`
|
||
- `void pumpLoads(ResourceManager&, FrameResources&)`
|
||
- `void evictToBudget(size_t bytes)`
|
||
- Controls: `set_max_loads_per_pump`, `set_keep_source_bytes`, `set_cpu_source_budget`, `set_gpu_budget_bytes`
|
||
|
||
Defaults & Budgets
|
||
- Worker threads: 1–4 decode threads depending on hardware.
|
||
- Loads per pump: default 4.
|
||
- Upload byte budget: default 128 MiB per frame.
|
||
- GPU budget: unlimited until the engine sets one each frame. The engine queries ~35% of device‑local memory (via VMA) and calls `set_gpu_budget_bytes(...)`, then runs `evictToBudget(...)` and `pumpLoads(...)` during the frame loop (`src/core/vk_engine.cpp`).
|
||
- CPU source bytes: default budget 64 MiB; `keep_source_bytes` defaults to false. Retention only applies to entries created from Bytes keys.
|
||
|
||
Examples
|
||
- Asset materials (`src/core/asset_manager.cpp`)
|
||
- Create materials with visible fallbacks (checkerboard/white/flat‑normal), then:
|
||
- Build a key from an asset path, `request(key, sampler)`, and `watchBinding(handle, materialSet, binding, sampler, fallbackView)` for albedo (1), metal‑rough (2), normal (3).
|
||
- glTF loader (`src/scene/vk_loader.cpp`)
|
||
- Builds keys from URI/Vector/BufferView sources, requests handles, and registers watches for material textures. On teardown, calls `unwatchSet(materialSet)` before resetting descriptor pools to avoid patching dead sets. The geometry/transparent passes mark used sets each frame.
|
||
|
||
Implementation Notes
|
||
- Uploads and layouts
|
||
- Deferred uploads: the RG transfer pass transitions `UNDEFINED → TRANSFER_DST_OPTIMAL`, copies, and either generates mipmaps (finishing in `SHADER_READ_ONLY_OPTIMAL`) or transitions directly there. No extra transition is needed after mip gen.
|
||
- Descriptor rewrites
|
||
- Material descriptor sets and pools are created with `UPDATE_AFTER_BIND` flags; patches are applied safely across frames using a `DescriptorWriter`.
|
||
- Key hashing
|
||
- 64‑bit FNV‑1a for dedup. FilePath keys hash `PATH:<path>#(sRGB|UNORM)`. Bytes keys hash the payload and XOR an sRGB tag when requested.
|
||
- Format selection and channel packing
|
||
- `TextureKey::channels` can be `Auto` (default), `R`, `RG`, or `RGBA`. The cache chooses `VK_FORMAT_R8/R8G8/RGBA8` (sRGB variants when requested) and packs channels on CPU for `R`/`RG` to reduce staging + VRAM.
|
||
- Progressive downscale
|
||
- The decode thread downsizes large images by powers of 2 until within `Max Upload Dimension`, reducing both staging and VRAM. You can increase the cap or disable it (set to 0) from the UI.
|
||
|
||
KTX2 specifics
|
||
- Supported: 2D, single‑face, single‑layer KTX2. If BasisLZ/UASTC, libktx transcodes to BCn. sRGB/UNORM is honored from the file’s DFD and can be nudged by request (albedo sRGB, MR/normal UNORM).
|
||
- Not supported: Cube/array/multilayer KTX2 (current code path assumes single layer, 2D).
|
||
|
||
Limitations / Future Work
|
||
- Linear‑blit capability check
|
||
- `generate_mipmaps` always uses `VK_FILTER_LINEAR`. Add a format/feature check and a fallback path (nearest or compute downsample).
|
||
- Texture formats
|
||
- Raster path: 8‑bit R/RG/RGBA via stb_image. Compressed path: BCn via `.ktx2`. Future: ASTC/ETC2, specialized R8/RG8 parsing, and float HDR support (`stbi_loadf` → `R16G16B16A16_SFLOAT`).
|
||
- Normal‑map mip quality
|
||
- Linear blits reduce normal length; consider a compute renormalization pass.
|
||
- Samplers
|
||
- Anisotropy is currently disabled in `SamplerManager`; enable when supported and expose a knob.
|
||
- Minor robustness
|
||
- `enqueue_decode()` derives the handle via pointer arithmetic on `_entries`. Passing the precomputed index would avoid any future reallocation hazards.
|
||
|
||
Operational Tips
|
||
- Keep deferred uploads enabled (`ResourceManager::set_deferred_uploads(true)`) to coalesce copies per frame (engine does this during init).
|
||
- To debug VMA allocations and name images, set `VE_VMA_DEBUG=1`.
|