Repository navigation
[FEATURE REQUEST] Use the project's WIKI to share results for each Ruby version with latest CI results #206
Description
Activity
I like the core concepts here, and think there might be an easier option:
Store the results as .md (and json/yaml) files in the filesystem, with the folder as the Ruby version number.
That would give the following benefits:
- The wiki could canonically link to the folder
- Since it could be stored as
results/version/README.md, github would display the rendered markdown when users navigate through - CI would only need to commit the relevant files
- Users could create tools to track differences between Ruby versions
We might be able to leverage benchmark.fyi which is an OSS Rails app that stores benchmark-ips results.
I've submitted a PR that shares the results to https://ips.fastruby.io (powered by benchmark.fyi) -- you can see it here: #203
One enhancement that I'd like to add to benchmark-ips and benchmark.fyi is more contextual information:
- Ruby version
- OS version
- Other info about the environment that was running the benchmark code?
Not sure how all of this would tie together, but maybe there could be a wiki page that keeps track of the benchmarks we run and share for this repo?
Just reading this issue I explain the sort of same fix in this comment: #205 (comment)
I will assign this to me, so I don't forget about this, to ping back when we sort this out
The results site is live: https://fastruby.github.io/fast-ruby/
It shows every benchmark on every Ruby CI runs (2.1 to head, JRuby, TruffleRuby, with and without YJIT and ZJIT), rebuilt from CI on every merge that runs the benchmarks (#250). We went with GitHub Pages instead of the wiki, so the numbers come straight from the CI runs.
@arielj could you have a look and tell us if this covers what you had in mind?
Currently, the results are in the README file, but all numbers are from one specific ruby version. Trying to add information for more rubies can make the README a mess and super large to follow.
Another source for the benchmark results is the result of CI runs which shows up-to-date and more rubies, but it's not formatted for readability, it's just the output of the rake command which is not ideal.
Maybe another option is to use the Wiki of the project creating one page with the results for each Ruby that's tested, and then use a github action to update the results there so each page is always automatically updated with the latest results from CI.
I've never done this but I think it's possible, there's this action for example that creates a Wiki page given a markdown file https://github.com/Decathlon/wiki-page-creator-action. So I understand it's possible to create/edit Wiki pages if we can output the benchmark results as a .md file using the github API.
This would help finding benchmarks relevant to the Ruby version the user is using, since different ruby versions may add new optimizations to some of the code variants and old Ruby benchmarks would be inaccurate.