feat(onboarding): Org-level onboarding experiment events - #8242
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe API now evaluates and serialises an organisation’s onboarding variant. The frontend selects the onboarding flow from Estimated code review effort: 4 (Complex) | ~45 minutes Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
front-end
The onboarding flow decision is now made in the backend, per organisation. Serving a multivariate split assignment records the experiment exposure; the first SDK evaluation of any of the organisation's environments records the conversion, both against the organisation targeting key. - flagsmith 6.2.0 and openfeature-provider-flagsmith from its experimentation-tracking PR branch - events enabled on the Flagsmith-on-Flagsmith client via FLAGSMITH_ON_FLAGSMITH_SERVER_EVENTS_API_URL (set in staging and production ECS task definitions; never enabled in offline mode) - get_openfeature_client() attaches the provider's exposure hook, so any multivariate split evaluation records an exposure - onboarding_variant field on the organisation serialiser beep boop
The gate no longer evaluates the Flagsmith-on-Flagsmith flag client-side; the backend decides per organisation and the exposure is recorded server-side. beep boop
74e6760 to
015d7fd
Compare
github-actions
Bot
added
feature
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8242 +/- ##
========================================
Coverage 98.72% 98.73%
========================================
Files 1558 1561 +3
Lines 61948 62048 +100
========================================
+ Hits 61160 61260 +100
Misses 788 788 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f621a0df-02a6-4b18-8da0-dc4dd25e31d7
⛔ Files ignored due to path filters (1)
api/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (24)
api/app/settings/common.pyapi/environments/onboarding/services.pyapi/integrations/flagsmith/client.pyapi/organisations/serializers.pyapi/organisations/services.pyapi/pyproject.tomlapi/tests/unit/environments/onboarding/test_unit_environments_onboarding_services.pyapi/tests/unit/integrations/flagsmith/test_unit_flagsmith_client.pyapi/tests/unit/organisations/test_unit_organisations_serializers.pyapi/tests/unit/organisations/test_unit_organisations_services.pydocs/docs/deployment-self-hosting/observability/_events-catalogue.mddocs/docs/integrating-with-flagsmith/CLI.mddocs/docs/integrating-with-flagsmith/legacy-cli.mddocs/docs/integrating-with-flagsmith/sdks/client-side-sdks/javascript.mddocs/docs/integrating-with-flagsmith/sdks/server-side.mdxfrontend/common/types/responses.tsfrontend/common/utils/getOnboardingVariant.tsfrontend/web/components/pages/onboarding/GettingStartedGate.tsxinfrastructure/aws/production/ecs-task-definition-admin-api.jsoninfrastructure/aws/production/ecs-task-definition-sdk-api.jsoninfrastructure/aws/production/ecs-task-definition-task-processor.jsoninfrastructure/aws/staging/ecs-task-definition-admin-api.jsoninfrastructure/aws/staging/ecs-task-definition-sdk-api.jsoninfrastructure/aws/staging/ecs-task-definition-task-processor.json
💤 Files with no reviewable changes (1)
- frontend/common/utils/getOnboardingVariant.ts
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 8
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f621a0df-02a6-4b18-8da0-dc4dd25e31d7
⛔ Files ignored due to path filters (1)
api/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (24)
api/app/settings/common.pyapi/environments/onboarding/services.pyapi/integrations/flagsmith/client.pyapi/organisations/serializers.pyapi/organisations/services.pyapi/pyproject.tomlapi/tests/unit/environments/onboarding/test_unit_environments_onboarding_services.pyapi/tests/unit/integrations/flagsmith/test_unit_flagsmith_client.pyapi/tests/unit/organisations/test_unit_organisations_serializers.pyapi/tests/unit/organisations/test_unit_organisations_services.pydocs/docs/deployment-self-hosting/observability/_events-catalogue.mddocs/docs/integrating-with-flagsmith/CLI.mddocs/docs/integrating-with-flagsmith/legacy-cli.mddocs/docs/integrating-with-flagsmith/sdks/client-side-sdks/javascript.mddocs/docs/integrating-with-flagsmith/sdks/server-side.mdxfrontend/common/types/responses.tsfrontend/common/utils/getOnboardingVariant.tsfrontend/web/components/pages/onboarding/GettingStartedGate.tsxinfrastructure/aws/production/ecs-task-definition-admin-api.jsoninfrastructure/aws/production/ecs-task-definition-sdk-api.jsoninfrastructure/aws/production/ecs-task-definition-task-processor.jsoninfrastructure/aws/staging/ecs-task-definition-admin-api.jsoninfrastructure/aws/staging/ecs-task-definition-sdk-api.jsoninfrastructure/aws/staging/ecs-task-definition-task-processor.json
💤 Files with no reviewable changes (1)
- frontend/common/utils/getOnboardingVariant.ts
🛑 Comments failed to post (3)
docs/docs/integrating-with-flagsmith/CLI.md (1)
31-40: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail curl -fsSL https://raw.githubusercontent.com/Flagsmith/flagsmith-cli/main/README.md | grep -nE 'pin.*install|release archive|install.sh'Repository: Flagsmith/flagsmith
Length of output: 647
Pin the installer source before making it the default installation path.
These copy-and-paste commands execute a mutable installer from the
mainbranch, which gives that script code execution on the user's machine if the repository or branch is compromised. Document the pinned installer as the recommended path and label themain-branch command as a quick-start or convenience option.Source: MCP tools
docs/docs/integrating-with-flagsmith/legacy-cli.md (1)
45-51: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail npx --yes `@flagsmith/cli` get --help | grep -E -- '--output|--api|--identity|ENVIRONMENT'Repository: Flagsmith/flagsmith
Length of output: 1180
🌐 Web query:
@flagsmith/cli legacy get command help ENVIRONMENT_API_KEY options --o -o💡 Result:
The Flagsmith CLI
getcommand is used to retrieve feature flags from the Flagsmith API and output them to a file [1][2]. TheENVIRONMENTargument represents the client-side environment key, which can be provided directly as an argument or via theFLAGSMITH_ENVIRONMENTenvironment variable [1][3]. The-o(or--output) option allows you to specify the destination file path for the output [1][2]. Usage syntax:flagsmith get [ENVIRONMENT] [-o <value>] [additional options][1][2] Available options for thegetcommand include: --o,--output=<value>: The file path to save the output (default:./flagsmith.json) [1][2]. --a,--api=<value>: The API URL to fetch feature flags from (default:https://edge.api.flagsmith.com/api/v1/) [1][2]. --i,--identity=<value>: The identity for which to fetch feature flags [1][2]. --e,--entity=<option>: Specify the entity to fetch, eitherflagsorenvironment(for local evaluation mode) [2]. --p,--pretty: Prettify the resulting JSON output [2]. Example usage:flagsmith get <ENVIRONMENT_API_KEY> -o ./my-flags.json[1][2] If you omit the environment key, the CLI will attempt to use the value set in theFLAGSMITH_ENVIRONMENTenvironment variable [1][3].Citations:
- 1: https://docs.flagsmith.com/integrating-with-flagsmith/CLI
- 2: https://github.com/Flagsmith/flagsmith-cli
- 3: https://docs.flagsmith.com/integrating-with-flagsmith/sdks/client-side-sdks/javascript
Make the examples match the legacy
getinterface.Use
<ENVIRONMENT_API_KEY>instead of<ENVIRONMENT_ID>, and use the declared short options-o,-a, and-irather than--o,--a, and--i.Suggested example changes
- $ flagsmith get <ENVIRONMENT_ID> + $ flagsmith get <ENVIRONMENT_API_KEY> - $ flagsmith get --o ./my-file.json + $ flagsmith get -o ./my-file.json - $ flagsmith get --a https://flagsmith.example.com/api/v1/ + $ flagsmith get -a https://flagsmith.example.com/api/v1/ - $ flagsmith get --i flagsmith_identity + $ flagsmith get -i flagsmith_identity📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.$ flagsmith get <ENVIRONMENT_API_KEY> $ flagsmith get -o ./my-file.json $ flagsmith get -a https://flagsmith.example.com/api/v1/ $ flagsmith get -i flagsmith_identitySource: MCP tools
docs/docs/integrating-with-flagsmith/sdks/server-side.mdx (1)
867-867: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail curl -fsSL https://raw.githubusercontent.com/Flagsmith/flagsmith-cli/main/README.md | grep -nE 'environment document|evaluate.*--js' curl -fsSL https://raw.githubusercontent.com/Flagsmith/flagsmith-nodejs-client/main/sdk/offline_handlers.ts | grep -nE 'buildEnvironmentModel|EnvironmentModel'Repository: Flagsmith/flagsmith
Length of output: 902
🏁 Script executed:
#!/usr/bin/env bash set -euo pipefail # Inspect the surrounding context and the referenced command in the documentation. sed -n '840,885p' docs/docs/integrating-with-flagsmith/sdks/server-side.mdx # Search for LocalFileHandler and flagsmith command references in the SDK docs/source files. rg -n "LocalFileHandler|flagsmith eval|--js|flagsmith environment document|environment document" docs docs/docs package.json -S || trueRepository: Flagsmith/flagsmith
Length of output: 16333
Use the environment-document command for
LocalFileHandler.
LocalFileHandlerreads a file and buildsEnvironmentModel.flagsmith eval --jswrites SDK resolution state for frontend SDKs, not the server-side local-evaluation environment document. The Node.js offline setup document soLocalFileHandleruses an invalid payload.Source: MCP tools
Also refreshes a stale line reference in the generated events catalogue. beep boop
Docker builds report
|
github-actions
Bot
added
feature
✅ private-cloud · depot-ubuntu-latest-arm-16 — run #19121 (attempt 1)Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)
Details
🗂️ Previous results✅ private-cloud · depot-ubuntu-latest-16 — run #19121 (attempt 1)Playwright Test Results (private-cloud - depot-ubuntu-latest-16)
Details
✅ oss · depot-ubuntu-latest-arm-16 — run #19121 (attempt 1)Playwright Test Results (oss - depot-ubuntu-latest-arm-16)
Details
✅ oss · depot-ubuntu-latest-16 — run #19121 (attempt 1)Playwright Test Results (oss - depot-ubuntu-latest-16)
Details
✅ private-cloud · depot-ubuntu-latest-arm-16 — run #19118 (attempt 1)Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)
Details
✅ private-cloud · depot-ubuntu-latest-16 — run #19117 (attempt 1)Playwright Test Results (private-cloud - depot-ubuntu-latest-16)
Details
✅ private-cloud · depot-ubuntu-latest-16 — run #19118 (attempt 1)Playwright Test Results (private-cloud - depot-ubuntu-latest-16)
Details
✅ oss · depot-ubuntu-latest-16 — run #19118 (attempt 1)Playwright Test Results (oss - depot-ubuntu-latest-16)
Details
✅ oss · depot-ubuntu-latest-arm-16 — run #19118 (attempt 1)Playwright Test Results (oss - depot-ubuntu-latest-arm-16)
Details
✅ oss · depot-ubuntu-latest-arm-16 — run #19117 (attempt 1)Playwright Test Results (oss - depot-ubuntu-latest-arm-16)
Details
|
Visual Regression19 screenshots compared. See report for details. |
github-actions
Bot
added
docs
| }, [organisation?.id, variant]) | ||
|
|
||
| return isSinglePageOnboarding() ? <OnboardingFlow /> : <GettingStartedPage /> | ||
| return variant === 'single_page' ? <OnboardingFlow /> : <GettingStartedPage /> |
There was a problem hiding this comment.
This decides on the org now, but App.js still decides on the flag in two places, so the two can disagree.
Whether a new user reaches /getting-started at all:
flagsmith/frontend/web/components/App.js
Lines 150 to 154 in 9d61fbe
And the flag that feeds the chromeless layout:
flagsmith/frontend/web/components/App.js
Lines 242 to 244 in 9d61fbe
flagsmith/frontend/web/components/App.js
Lines 318 to 323 in 9d61fbe
These need to read onboarding_variant as well, otherwise the result might vary.
There was a problem hiding this comment.
Thanks so much for handling that @khvn26. 🙏 😃
Just a few comments regarding FE part
The E2E tests seems to still skip on the frontend flag, but the API decides the flow now. Could you double check they still do the right thing or can be adjusted?
flagsmith/frontend/e2e/tests/onboarding-tests.pw.ts
Lines 16 to 19 in 9d61fbe
flagsmith/frontend/e2e/tests/initialise-tests.pw.ts
Lines 20 to 23 in 9d61fbe
4 participants
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Contributes to #8153.
The onboarding experiment splits on organisations, with both exposure and conversion experiment events emitted by the backend.
/api/v1/organisations/now returnsonboarding_variant, and the frontend decides which flow to render based on that.The Core API now has a
FLAGSMITH_ON_FLAGSMITH_SERVER_EVENTS_API_URLsetting. If FoF not in offline mode and the new setting is set, the SDK is initialised with event tracking capability.How did you test this code?
Added unit tests.