Skip to content

Introduce skid_steer as a new way to load assets - #515

Open
patowen wants to merge 6 commits into
Ralith:masterfrom
patowen:introduce-skid-steer
Open

Introduce skid_steer as a new way to load assets#515
patowen wants to merge 6 commits into
Ralith:masterfrom
patowen:introduce-skid-steer

Conversation

@patowen

@patowen patowen commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

This is a relatively significant refactor of Hypermine, switching up asset loading to use a new library called "skid_steer".

It introduces the following changes:

  • A custom AssetLoader type is added to drive asset loading, acting as the glue between skid_steer and Hypermine. This completely replaces the old loader module.
  • All uses of the staging buffer and transfer module from the temporary lahar_deprecated module have been replaced with a vendored growable_ring implementation from @Ralith, and a ParallelQueue from the current version of lahar. This allows us to finally remove the lahar_deprecated module, a task that I had hoped to do years ago.
  • PNG loading and GLTF loading, the only two processes that depended on the old Loader, have been migrated to the new AssetLoader.
  • File-processing (reading PNGs and GLBs) and Vulkan-wrangling (making Vulkan calls) have been fully separated, with all data being read into memory in a convenient form before being passed into Vulkan data structures.
  • To support some of the above changes, some data has been moved around, including the addition of a new ShaderData struct to pass around global data used for rendering.

@patowen
patowen force-pushed the introduce-skid-steer branch 2 times, most recently from 7eb271f to 614549c Compare August 21, 2026 03:05
@patowen
patowen force-pushed the introduce-skid-steer branch from 614549c to 656444b Compare August 22, 2026 02:41
.image(handle)
.subresource_range(range)],
);
xf.device.cmd_clear_color_image(

@patowen patowen Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When separating out gtlf-reading logic with Vulkan wrangling, I had decided to remove the functionality that uses cmd_clear_color_image to fill the 1x1 image with a specified color, in favor of just specifying the color directly.

However, this requires a conversion from f32 color to SRGB, which you might have opinions on how to do. One option could be to import something like https://crates.io/crates/palette (although you might have a preferred crate for this). Given that we might want to do more in the future, I don't think we want to hardcode the gamma correction formula.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gotten to the context of this question yet, but the color crate is an alternative to consider for color space conversions; it's maintained by some extremely reputable 2D graphics folks. I don't think we need an external crate for gamma conversion but it's fine to use one.

Comment thread client/src/graphics/gltf_mesh.rs Outdated
.material()
.pbr_metallic_roughness()
.base_color_factor()
.map(|c| 255) // TODO: Will likely want a crate for color conversion

@patowen patowen Aug 22, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://github.com/Ralith/hypermine/pull/515/changes/BASE..0908719a3be8e59af41e30e075cbe97f0001cf17#r3834892686.

I deliberately kept the lint failing due to this line so that I wouldn't forget to address this.

EDIT: Actually, now that there's an unresolved PR comment, this shouldn't be forgotten anyway. I'll fix the lint.

@patowen
patowen force-pushed the introduce-skid-steer branch from 2ac99ab to 359c1aa Compare August 25, 2026 02:21
@patowen
patowen marked this pull request as ready for review August 26, 2026 02:39

@Ralith Ralith left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review is just getting started, but it may be a few days before I make more progress.

}

fn get_all_events(&mut self) -> Vec<Event> {
let events = self.events.lock().unwrap().clone();

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: &mut self means we have exclusive access, so there's no need to mess around with locking. Prefer get_mut.

/// Drains all events that were returned by the most recent call to get_all_events
fn drain_queried_events(&mut self) {
self.events
.lock()

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

}
}

fn test_eventual_success(mut f: impl FnMut() -> Result<(), anyhow::Error>) {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is complex and may still be fragile; there's no hard guarantee on a desktop OS that any particular thread will be scheduled in any particular time interval. Prefer to actually wait for the event we're interested in. For example, you could use a std::sync::Condvar to implement a helper that blocks until an event is delivered, and call that in a loop.

impl AssetLoadContext {
/// # Safety
/// - [`Work::cmd`] must not be used outside the lifetime of the returned [`Work`]
/// - Any Vulkan resources this work uses must not be destroyed before the [`Work`] is dropped

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely they must outlive the execution of the work?


/// Timeline sempahores must be signaled in a strictly increasing sequence, so having multiple threads manage
/// the semaphore that unparks the timeline queue is error-prone. The purpose of this thread is to centralize
/// management of this semaphore.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need to trigger this from multiple threads? If so, would it be simpler to use a Mutex<u64> that we hold across the signal operation?

Comment thread client/src/graphics/asset_loader.rs Outdated
}

#[test]
#[cfg_attr(not(feature = "run_vulkan_tests"), ignore)]

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just install lavapipe and maintain full test coverage in CI.

.expect("runtime using this context should already be dropped");

// Fail-safe to ensure that the parallel queue is driven at least once after all work has been submitted before
// we drain the handle

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would that be needed? If we don't intend for it to be needed, then this will prevent us from detecting bugs with tests.

) {
loop {
// Systems increment the `queue_unpark_semaphore` value when they want to guarantee
// that we don't park unless certain things are done. `ParallelQueueWaiter`

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is ParallelQueueWaiter?

pub fn queue_family(&self) -> u32 {
self.gfx.queue_family
pub async fn wait_for_completion(&self, semaphore_value: u64) {
// To actually get the work to start, we need to unpark the queue. We do it here to avoid getting stuck awaiting something we never kicked off.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed? Submitting work should unpark the queue automatically.

queue_unpark_semaphore: vk::Semaphore,
shutdown_token: CancellationToken,
queue_watch_sender: &tokio::sync::watch::Sender<u64>,
staging: &GrowableRing,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider aggregating this stuff into a struct with a fn run(self).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants