## Picking System: Object Selection and Interaction Unified picking system that handles single-click selection, hover detection, and drag-box multi-select. Supports both CPU ray-casting (via BVH) and GPU ID-buffer picking. ### Components - `PickingSystem` (src/core/picking/picking_system.h/.cpp) - Main entry point for all picking operations. - Processes SDL mouse events (click, drag, release). - Maintains per-frame hover picks and last click selection. - Integrates with RenderGraph for async ID-buffer readback. - `SceneManager` picking helpers (src/scene/vk_scene_picking.cpp) - CPU ray-casting against `RenderObject` bounds. - BVH-accelerated mesh picking for precise triangle-level hits. - Rectangle selection in NDC space. ### PickInfo Structure Result of any pick operation: ```cpp struct PickInfo { MeshAsset *mesh = nullptr; // Source mesh asset LoadedGLTF *scene = nullptr; // Source glTF scene Node *node = nullptr; // glTF node that owns this surface RenderObject::OwnerType ownerType; // StaticGLTF, GLTFInstance, MeshInstance std::string ownerName; // Logical name (e.g., "player", "cube01") WorldVec3 worldPos; // Hit position in world space (double-precision) glm::mat4 worldTransform; // Object's world transform uint32_t indexCount; // Index count of picked surface uint32_t firstIndex; // First index of picked surface uint32_t surfaceIndex; // Surface index within mesh bool valid = false; // True if pick hit something }; ``` ### Picking Modes #### 1. CPU Ray Picking (Default) Uses camera ray against object bounds with BVH acceleration: - **Sphere bounds**: Quick bounding sphere test. - **Box bounds**: Ray-box intersection in local space. - **Capsule bounds**: Combined cylinder + sphere caps intersection. - **Mesh bounds**: Full BVH traversal for triangle-level precision. Advantages: - Immediate results (same frame). - No GPU readback latency. - Precise triangle-level hits with mesh BVH. #### 2. ID-Buffer Picking Renders object IDs to a GPU buffer and reads back single pixel: - Each `RenderObject` has a unique `objectID` assigned during draw. - Geometry pass writes IDs to an R32_UINT attachment. - `PickReadback` render pass copies single pixel to CPU-readable buffer. - Result resolved on next frame after GPU fence wait. Advantages: - Pixel-perfect accuracy (no false positives from bounds). - Handles complex overlapping geometry. - Works with any object shape. Enable via: ```cpp picking_system.set_use_id_buffer_picking(true); ``` ### PickingSystem API **Initialization:** ```cpp void init(EngineContext *context); void cleanup(); ``` **Frame Lifecycle:** ```cpp void begin_frame(); // Resolve pending async picks after fence wait void process_event(const SDL_Event &event, bool ui_want_capture_mouse); void update_hover(); // Update hover pick each frame ``` **Results Access:** ```cpp const PickInfo& last_pick() const; // Last click selection const PickInfo& hover_pick() const; // Current hover (under cursor) const std::vector& drag_selection() const; // Multi-select results uint32_t last_pick_object_id() const; // Raw object ID of last pick ``` **Configuration:** ```cpp bool use_id_buffer_picking() const; void set_use_id_buffer_picking(bool enabled); bool debug_draw_bvh() const; void set_debug_draw_bvh(bool enabled); ``` **Utilities:** ```cpp // Clear picks for removed objects (call when deleting instances) void clear_owner_picks(RenderObject::OwnerType owner_type, const std::string &owner_name); // Mutable access for engine integration PickInfo* mutable_last_pick(); PickInfo* mutable_hover_pick(); ``` **RenderGraph Integration:** ```cpp // Called during RenderGraph build when ID-buffer picking is enabled void register_id_buffer_readback(RenderGraph &graph, RGImageHandle id_buffer, VkExtent2D draw_extent, VkExtent2D swapchain_extent); ``` ### SceneManager Picking API Lower-level picking functions for custom use: ```cpp // Single-object ray pick bool pick(const glm::vec2 &mousePosPixels, RenderObject &outObject, WorldVec3 &outWorldPos); // Resolve ID from ID-buffer back to RenderObject bool resolveObjectID(uint32_t id, RenderObject &outObject) const; // Rectangle selection (drag-box) void selectRect(const glm::vec2 &p0, const glm::vec2 &p1, std::vector &outObjects) const; ``` ### Usage Examples **Basic Click Selection:** ```cpp void Game::handle_selection(PickingSystem& picking) { const PickingSystem::PickInfo& pick = picking.last_pick(); if (pick.valid) { fmt::println("Selected: {} at ({}, {}, {})", pick.ownerName, pick.worldPos.x, pick.worldPos.y, pick.worldPos.z); // Handle different owner types switch (pick.ownerType) { case RenderObject::OwnerType::GLTFInstance: select_actor(pick.ownerName); break; case RenderObject::OwnerType::MeshInstance: select_prop(pick.ownerName); break; default: break; } } } ``` **Hover Tooltips:** ```cpp void Game::update_ui(PickingSystem& picking) { const PickingSystem::PickInfo& hover = picking.hover_pick(); if (hover.valid) { show_tooltip(hover.ownerName); } } ``` **Multi-Select with Drag Box:** ```cpp void Game::process_drag_selection(PickingSystem& picking) { const auto& selection = picking.drag_selection(); for (const PickingSystem::PickInfo& pick : selection) { if (pick.valid) { add_to_selection(pick.ownerName); } } } ``` **Custom Ray Pick:** ```cpp void Game::custom_pick(SceneManager& scene, const glm::vec2& screen_pos) { RenderObject hit_object{}; WorldVec3 hit_pos{}; if (scene.pick(screen_pos, hit_object, hit_pos)) { // hit_object contains the picked render object // hit_pos is the precise world-space hit position spawn_effect_at(hit_pos); } } ``` ### Bounds Types Objects can use different bounds types for picking: ```cpp enum class BoundsType : uint8_t { None = 0, // Not pickable Box = 1, // AABB (default for most objects) Sphere = 2, // Bounding sphere Capsule = 3, // Cylinder + hemisphere caps Mesh = 4, // Full BVH mesh intersection }; ``` Set bounds type when adding instances: ```cpp scene->addMeshInstance("capsule_enemy", mesh, transform, BoundsType::Capsule); ``` ### Mesh BVH Picking For precise triangle-level picking on complex meshes: 1. Mesh assets automatically build a BVH during loading. 2. `BoundsType::Mesh` triggers BVH traversal instead of simple bounds test. 3. Returns exact triangle hit position, not just bounding box intersection. 4. `PickInfo::firstIndex` and `indexCount` are refined to the exact triangle. Debug BVH visualization: ```cpp picking.set_debug_draw_bvh(true); // Shows BVH nodes in debug overlay ``` ### Coordinate Space Handling The picking system handles multiple coordinate transformations: 1. **Window pixels** (SDL event coordinates, top-left origin) 2. **Swapchain pixels** (scaled for HiDPI displays) 3. **Logical render pixels** (internal render resolution) 4. **NDC** (normalized device coordinates, -1 to 1) 5. **World space** (double-precision `WorldVec3`) The `window_to_swapchain_pixels()` helper handles HiDPI scaling. Letterboxing is accounted for when render resolution differs from swapchain size. ### Frame Flow 1. `begin_frame()` — Resolve pending async ID-buffer picks from previous frame. 2. `process_event()` — Handle mouse events (click start/end, motion). 3. `update_hover()` — CPU ray-cast for current hover under cursor. 4. RenderGraph build — If ID-buffer picking enabled, register readback pass. 5. Next frame — Async pick result becomes available. ### Integration with ImGui The picking system respects ImGui's input capture: ```cpp // In event loop bool ui_capture = imgui_system.want_capture_mouse(); picking_system.process_event(event, ui_capture); ``` When `ui_want_capture_mouse` is true: - Click events are ignored (no picks started). - Mouse motion still updates cursor position for future picks. - Hover picking is not affected. ### Tips - Use CPU ray picking (`set_use_id_buffer_picking(false)`) for immediate feedback. - Use ID-buffer picking for pixel-perfect selection in dense scenes. - Call `clear_owner_picks()` when removing instances to avoid stale picks. - For mesh BVH to work, ensure the mesh was loaded with BVH generation enabled. - The drag selection threshold is 3 pixels — smaller motions are treated as clicks.