Allow a signed object storage folder as the origin

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Breaking:** a path in `publisher.origin_url` is kept, and the request path is added after it with one `/`. The path was discarded before, so a deployment whose `origin_url` has one now sends each request to the folder the path names, and answers `400` to a request path with a `.` or `..` segment, with or without `origin_auth`. A deployment whose `origin_url` has no path sends its requests as before. Before upgrading, check that the origin serves the site at that path, or remove the path.
+- **Breaking:** the check that closes `/_ts/admin` and `/admin/keys` on every adapter now also answers `404` to a request path still changing after four rounds of percent-decoding and to a path that reaches either once an escape that is not valid UTF-8 is replaced. The publisher received both before, and no setting turns the check off.
 - **Breaking for crates outside core:** a geo module is a `GeoModule`, called with the request's module call as a device module is (`locate(call)`), so it sees the request's evidence, the settings and the services. `RuntimeServicesBuilder::geo`, `RuntimeServices::with_geo` and `IntegrationRegistrationBuilder::with_geo_module` take an `Arc<dyn GeoModule>`, and `RuntimeServices::geo()`, `IntegrationRegistry::geo_module()`, the `IntegrationRegistration::geo_module` field and `build_geo_module` return or hold one. A module that resolves from the client address alone keeps implementing `PlatformGeo` and is offered through `PlatformGeoModule::new(name, module)`, which hands the platform lookup the address the call carries. Core's lookups on the Edge Cookie and auction paths build the call from the request.
 - **Breaking:** no response carries an `x-geo-*` header, `x-geo-info-available` included, unless the deployment selects `headers.geo` (`[headers] modules = ["geo"]`) and names it in a `[[serve]]` entry. That serve middleware writes the location already resolved for the request on the responses the entry covers, and no adapter calls a geo module or writes the headers when it finishes a response. For crates outside core, `GeoInfo::set_response_headers` and the `HEADER_X_GEO_*` constants are gone from core.
 - The Axum dev server and Spin, which have no geo lookup, refuse `[geo] module = "platform"` at startup naming the setting. The selection located no request there and still counted as a geo module, so a deployment running an Edge Cookie module was never asked to set `[geo] assume_single_jurisdiction`. The configuration and Cloudflare guides say that Cloudflare places a visitor in a region only from the `cf-region-code` request header.
@@ -97,6 +99,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- A publisher's site may be served from a folder of object storage through the publisher proxy, so its pages pass through the page pipeline, which an asset route skips. The folder is the path of `origin_url`, and two optional `[publisher]` settings do the rest, each off unless set. `origin_auth` takes the table an asset route's `auth` takes and signs every origin request with it, through the same signer. `directory_index` names a file added to a request path ending in `/`, for the origin request only. A signed origin is only read, so a method other than `GET` or `HEAD` is answered `405`, and a signed request carries none of the reader's cookies, credentials, referrer, conditional validators or range. A malformed `origin_auth` or `directory_index` is refused when the settings load, with the setting named.
 - `[ad-tag.google] write_loader` writes the loader tag, for a page that carries no loader of its own, through the serve middleware `ad-tag.google.loader`, which a `[[serve]]` entry names and `ts audit` writes where it finds GPT. The tag goes in the head after the bootstrap and asks for the loader on the module's own route. Only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets it, because GPT loaded from Google's standard address may use the device's storage at any time. That is TCF Purpose 1, which core's server-side auction requires where GDPR applies and which the Google tag module's loaders require. Off by default, because a page that already carries the loader would otherwise load it twice.
 - The publisher's assets and the answers of the asset routes are held through the loading cache, selected in `[cache.assets]`, so readers who share a memory copy, or on Fastly a location, share one fetch of an asset, and the copy in the host's shared store answers later requests until the origin's own lifetime ends. A memory copy lasts one request on Cloudflare and Spin, and their key-value stores make no request wait, so there each request that misses at the same moment fetches the asset. A copy is kept under what the origin says it varies on, served with the headers that describe it and an `Age` that counts from the origin's, and nothing is kept that may not be shared. A request that sends the publisher's origin a cookie always goes to the origin, and its answer is not kept. A deployment that names no module takes the one loading cache module the build links, and `module = "none"`, or a build that links none, holds nothing, as before. A `[cache.assets]` lifetime outside the ranges of `[cache.robots-txt]` is refused when the settings load.
 - A Google tag module, `[tag] modules = ["google-tag"]` with `[tag.google-tag]`, from `crates/tag/google-tag`, which writes the consent mode defaults, Tag Manager's container script, the gtag.js snippet and the AdSense loader in the head immediately before the script bundle's tag, and Tag Manager's noscript iframe after the opening body tag, each copied from Google's documentation. Every reader gets the consent defaults, and only a reader whose permissions allow storage on the device (`necessary.operations.storage`) gets the loaders and the noscript iframe. The consent defaults come first, the four consent types Google requires for its own services default to denied, the tag waits `consent_defaults.wait_for_update` milliseconds (500 unless set) for a consent management platform's update before it fires with them, and `consent_defaults.region` scopes them to country or region codes, with every other visitor given all four denied. Where the Tag Manager module serves the same tag or container the loader names the first-party path, and a page's own loader for the configured tag is removed for every reader.

crates/trusted-server-core/src/closed_paths.rs

@@ -7,8 +7,6 @@
 //! every adapter asks [`closed_path_response`] before its publisher fallback
 //! and the request is answered `404` here.
 
-use std::borrow::Cow;
-
 use edgezero_core::body::Body as EdgeBody;
 use http::{Request, Response, StatusCode, header};
 
@@ -50,30 +48,28 @@ pub fn closed_path_response(req: &Request<EdgeBody>) -> Option<Response<EdgeBody
 /// decode as `/admin%2Fkeys/rotate`, so the check is repeated to a fixed point
 /// rather than applied once.
 pub(crate) fn is_closed(path: &str) -> bool {
-    if is_closed_as_written(path) {
-        return true;
-    }
+    matches_any_decoding(path, is_closed_as_written)
+}
 
+/// Returns whether `matches` holds for `path` as written or after any of
+/// [`MAX_PERCENT_DECODE_ROUNDS`] rounds of percent-decoding.
+///
+/// A path still changing after the last round counts as a match, so a check
+/// built on this fails closed.
+pub(crate) fn matches_any_decoding(path: &str, matches: impl Fn(&str) -> bool) -> bool {
     let mut current = path.to_owned();
-    for _ in 0..MAX_PERCENT_DECODE_ROUNDS {
-        let Some(decoded) = percent_decoded_path(&current) else {
-            return false;
-        };
-
-        // A path whose remaining `%` sequences are not decodable escapes is a
-        // fixed point, so further rounds would repeat the same comparison.
+    for _ in 0..=MAX_PERCENT_DECODE_ROUNDS {
+        if matches(&current) {
+            return true;
+        }
+        let decoded =
+            String::from_utf8_lossy(&urlencoding::decode_binary(current.as_bytes())).into_owned();
         if decoded == current {
             return false;
         }
-
-        if is_closed_as_written(&decoded) {
-            return true;
-        }
-
         current = decoded;
     }
-
-    false
+    true
 }
 
 fn is_closed_as_written(path: &str) -> bool {
@@ -84,16 +80,6 @@ fn is_closed_as_written(path: &str) -> bool {
             .is_some_and(|remainder| remainder.starts_with('/'))
 }
 
-/// Percent-decodes `path` once, returning `None` when the path contains no
-/// escape sequence or decodes to invalid UTF-8.
-fn percent_decoded_path(path: &str) -> Option<String> {
-    if !path.contains('%') {
-        return None;
-    }
-
-    urlencoding::decode(path).ok().map(Cow::into_owned)
-}
-
 fn not_found() -> Response<EdgeBody> {
     Response::builder()
         .status(StatusCode::NOT_FOUND)
@@ -216,6 +202,11 @@ mod tests {
         }
     }
 
+    #[test]
+    fn a_path_still_decoding_after_the_last_round_is_closed() {
+        assert_closed("/admin%2525252525252Fkeys/rotate");
+    }
+
     #[test]
     fn other_publisher_paths_are_left_to_the_fallback() {
         for path in [

crates/trusted-server-core/src/config.rs

@@ -279,37 +279,32 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig {
                 ],
                 true,
             ),
+        ];
+        // An asset route's `auth` and `[publisher]`'s `origin_auth` take the
+        // same table, so each holds the same three key references.
+        let origin_auth_keys = ["access_key_id", "secret_access_key", "session_token"];
+        fields.extend(origin_auth_keys.map(|key| {
             field(
                 vec![
                     optional_object("proxy"),
                     optional_object("asset_routes"),
                     SecretPathSegment::ArrayEach,
                     optional_object("auth"),
-                    object("access_key_id"),
-                ],
-                true,
-            ),
-            field(
-                vec![
-                    optional_object("proxy"),
-                    optional_object("asset_routes"),
-                    SecretPathSegment::ArrayEach,
-                    optional_object("auth"),
-                    object("secret_access_key"),
+                    object(key),
                 ],
                 true,
-            ),
+            )
+        }));
+        fields.extend(origin_auth_keys.map(|key| {
             field(
                 vec![
-                    optional_object("proxy"),
-                    optional_object("asset_routes"),
-                    SecretPathSegment::ArrayEach,
-                    optional_object("auth"),
-                    object("session_token"),
+                    object("publisher"),
+                    optional_object("origin_auth"),
+                    object(key),
                 ],
                 true,
-            ),
-        ];
+            )
+        }));
         // The settings a module declares in its own table, for core's
         // modules and the ones a tool registered for deploy validation.
         let builders = crate::integrations::all_builders(deploy_integrations()).collect::<Vec<_>>();
@@ -551,28 +546,40 @@ fn validate_secret_key_references(
     }
 
     for (index, route) in settings.proxy.asset_routes.iter().enumerate() {
-        let Some(AssetOriginAuth::S3SigV4(auth)) = route.auth.as_ref() else {
-            continue;
-        };
-        validate_secret_key_reference(
-            &format!("proxy.asset_routes[{index}].auth.access_key_id"),
-            auth.access_key_id.expose(),
-        )?;
-        validate_secret_key_reference(
-            &format!("proxy.asset_routes[{index}].auth.secret_access_key"),
-            auth.secret_access_key.expose(),
-        )?;
-        if let Some(token) = &auth.session_token {
-            validate_secret_key_reference(
-                &format!("proxy.asset_routes[{index}].auth.session_token"),
-                token.expose(),
+        if let Some(auth) = route.auth.as_ref() {
+            validate_origin_auth_key_references(
+                &format!("proxy.asset_routes[{index}].auth"),
+                auth,
             )?;
         }
     }
+    if let Some(auth) = settings.publisher.origin_auth.as_ref() {
+        validate_origin_auth_key_references("publisher.origin_auth", auth)?;
+    }
 
     Ok(())
 }
 
+/// Refuses an empty secret key reference in an origin auth table at `path`.
+fn validate_origin_auth_key_references(
+    path: &str,
+    auth: &AssetOriginAuth,
+) -> Result<(), Report<TrustedServerError>> {
+    let AssetOriginAuth::S3SigV4(auth) = auth;
+    validate_secret_key_reference(
+        &format!("{path}.access_key_id"),
+        auth.access_key_id.expose(),
+    )?;
+    validate_secret_key_reference(
+        &format!("{path}.secret_access_key"),
+        auth.secret_access_key.expose(),
+    )?;
+    if let Some(token) = &auth.session_token {
+        validate_secret_key_reference(&format!("{path}.session_token"), token.expose())?;
+    }
+    Ok(())
+}
+
 fn validate_secret_key_reference(
     path: &str,
     key_name: &str,
@@ -1044,6 +1051,9 @@ formats = [{ width = 300, height = 250 }]
                     true,
                 ),
                 ("proxy.asset_routes[*].auth.session_token".to_owned(), true),
+                ("publisher.origin_auth.access_key_id".to_owned(), true),
+                ("publisher.origin_auth.secret_access_key".to_owned(), true),
+                ("publisher.origin_auth.session_token".to_owned(), true),
             ],
             "should expose the native EdgeZero secret metadata contract"
         );
@@ -1398,6 +1408,94 @@ passphrase = "production-secret-key-32-bytes-min"
         );
     }
 
+    /// Valid settings whose `[publisher]` signs its origin requests.
+    fn settings_signing_the_publisher_origin(
+        access_key_id: &str,
+        secret_access_key: &str,
+        session_token: Option<&str>,
+    ) -> Settings {
+        let mut settings = valid_settings();
+        settings.publisher.origin_url = "https://storage.example.com/bucket/site".to_string();
+        settings.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig {
+            region: "test-region-1".to_string(),
+            secret_store: None,
+            access_key_id: Redacted::new(access_key_id.to_string()),
+            secret_access_key: Redacted::new(secret_access_key.to_string()),
+            session_token: session_token.map(|token| Redacted::new(token.to_string())),
+            origin_query: None,
+        }));
+        settings
+    }
+
+    #[test]
+    fn deploy_validation_accepts_a_publisher_origin_auth_naming_its_keys() {
+        validate_settings_for_deploy(&settings_signing_the_publisher_origin(
+            "origin_access_key_id",
+            "origin_secret_access_key",
+            Some("origin_session_token"),
+        ))
+        .expect("should accept an origin_auth whose key references are named");
+    }
+
+    #[test]
+    fn deploy_validation_refuses_an_empty_publisher_origin_auth_key_reference() {
+        for (settings, path) in [
+            (
+                settings_signing_the_publisher_origin(" ", "origin_secret_access_key", None),
+                "publisher.origin_auth.access_key_id",
+            ),
+            (
+                settings_signing_the_publisher_origin("origin_access_key_id", "", None),
+                "publisher.origin_auth.secret_access_key",
+            ),
+            (
+                settings_signing_the_publisher_origin(
+                    "origin_access_key_id",
+                    "origin_secret_access_key",
+                    Some(" "),
+                ),
+                "publisher.origin_auth.session_token",
+            ),
+        ] {
+            let err = validate_settings_for_deploy(&settings)
+                .expect_err("should refuse an empty origin_auth key reference");
+            assert!(
+                format!("{err:?}").contains(&format!(
+                    "secret key reference at `{path}` must not be empty"
+                )),
+                "should name `{path}`: {err:?}"
+            );
+        }
+    }
+
+    #[test]
+    fn documented_publisher_object_storage_settings_validate_when_uncommented() {
+        let toml = uncomment_block(
+            &template_with_resolved_required_secrets(),
+            "[publisher.origin_auth]",
+        )
+        .replacen(
+            "# directory_index = \"index.html\"",
+            "directory_index = \"index.html\"",
+            1,
+        );
+        let settings = Settings::from_toml(&toml)
+            .expect("the documented origin_auth and directory_index should parse and validate");
+
+        assert!(
+            matches!(
+                settings.publisher.origin_auth,
+                Some(AssetOriginAuth::S3SigV4(_))
+            ),
+            "the documented origin_auth should be read"
+        );
+        assert_eq!(
+            settings.publisher.directory_index.as_deref(),
+            Some("index.html"),
+            "the documented directory_index should be read"
+        );
+    }
+
     /// A table written for a module its section does not select is refused
     /// when the settings load, so `ts config validate` reports it before the
     /// configuration reaches a deployment.

crates/trusted-server-core/src/config_payload.rs

@@ -775,6 +775,44 @@ mod tests {
         );
     }
 
+    #[test]
+    fn publisher_origin_auth_resolves_its_credentials_from_the_store() {
+        let mut original = test_settings();
+        original.publisher.origin_url = "https://storage.example.com/bucket/site".to_string();
+        original.publisher.origin_auth = Some(AssetOriginAuth::S3SigV4(S3SigV4AuthConfig {
+            region: "test-region-1".to_string(),
+            secret_store: None,
+            access_key_id: Redacted::new("s3-access-key".to_string()),
+            secret_access_key: Redacted::new("s3-secret-key".to_string()),
+            session_token: Some(Redacted::new("s3-session-key".to_string())),
+            origin_query: None,
+        }));
+
+        let reconstructed = settings_from_config_blob(
+            &envelope_json(&original),
+            &UnifiedSecretStore,
+            &StoreName::from("ts_secrets"),
+        )
+        .expect("should resolve the publisher origin credentials from the mapped store");
+
+        let Some(AssetOriginAuth::S3SigV4(auth)) = reconstructed.publisher.origin_auth.as_ref()
+        else {
+            panic!("should keep the publisher origin_auth");
+        };
+        assert_eq!(auth.access_key_id.expose(), "AKIAIOSFODNN7EXAMPLE");
+        assert_eq!(
+            auth.secret_access_key.expose(),
+            "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
+        );
+        assert_eq!(
+            auth.session_token
+                .as_ref()
+                .map(Redacted::expose)
+                .map(String::as_str),
+            Some("resolved-session-token")
+        );
+    }
+
     #[test]
     fn partner_without_api_token_loads_without_secret_resolution() {
         let mut original = test_settings();

crates/trusted-server-core/src/inspect/config/tests.rs

@@ -208,6 +208,13 @@ fn a_key_the_inspect_section_does_not_have_is_refused_naming_it() {
 fn documents_holding_every_declared_secret() -> Vec<(&'static str, Value)> {
     let mut main = document("");
     main["publisher"]["proxy_secret"] = json!("proxy");
+    main["publisher"]["origin_auth"] = json!({
+        "type": "s3_sigv4",
+        "region": "test-region-1",
+        "access_key_id": "origin-access",
+        "secret_access_key": "origin-secret",
+        "session_token": "origin-session",
+    });
     main["ec"]["hmac"]["passphrase"] = json!("hmac");
     main["ec"]["partners"] = json!([{
         "name": "Example Partner",