feat: manage BYODS datasource lifecycle (#35)

* feat(byods): manage datasource lifecycle

* docs(byods): explain automatic datasource setup

---------

Co-authored-by: Adam Weeks <adweeks+cisco@cisco.com>

Author: adamweeks

AGENTS.MD

@@ -80,6 +80,7 @@ webex-byova-gateway-python/
 │   │   ├── i_vendor_connector.py
 │   │   └── local_audio_connector.py
 │   ├── core/                # Core gateway components
+│   │   ├── datasource_lifecycle.py
 │   │   ├── virtual_agent_router.py
 │   │   ├── wxcc_gateway_server.py
 │   │   └── *.py            # Generated gRPC stubs
@@ -138,6 +139,15 @@ webex-byova-gateway-python/
 - Converts between gRPC and connector formats
 - Handles error conditions and conversation cleanup
 
+### DataSourceLifecycle (`datasource_lifecycle.py`)
+
+- Uses the `webex-byods-sdk` package rather than implementing BYODS REST calls
+- Discovers or registers the gateway datasource before gRPC starts accepting traffic
+- Reconciles URL, schema, audience, subject, and active status at startup
+- Renews the datasource JWS before `tokenExpiryTime` and retries transient failures
+- Reads credential values only from explicitly named environment variables
+- Requires an explicit datasource ID when discovery returns multiple matches
+
 ## Development Guidelines
 
 ### Educational and Reference Purpose

README.md

@@ -27,6 +27,7 @@ The sample includes:
 
 - A BYOVA gRPC server with `ListVirtualAgents` and `ProcessCallerInput`
 - JWT validation for the WxCC data plane
+- Optional BYODS datasource registration and pre-expiry JWS renewal
 - A configuration-driven connector router
 - Local audio and AWS Lex connectors
 - gRPC and HTTP health checks
@@ -53,7 +54,8 @@ Before WxCC can connect to this gateway, you need:
   exchange domain for your gateway
 - Authorization of that Service App by an administrator in the target organization
 - A publicly reachable TLS-enabled gRPC server URL on the authorized domain
-- An `ACTIVE` BYOVA data-source registration whose URL exactly matches the public server URL
+- An `ACTIVE` BYOVA data-source registration whose URL exactly matches the public server URL;
+  the gateway can create and renew this registration when datasource management is enabled
 - A Contact Center AI virtual-agent configuration and a test flow that uses the Virtual
   Agent V2 activity
 - A configured gateway connector: use the local audio connector for a vendor-neutral
@@ -107,6 +109,30 @@ Never use disabled authentication for a Webex-connected or production endpoint.
 end-to-end test, configure the exact registered datasource URL and keep JWT enforcement
 enabled.
 
+### Configure Automatic Datasource Management
+
+The optional datasource lifecycle uses
+[`webex-byods-sdk`](https://github.com/WebexCommunity/webex-python-byods-sdk) to discover or
+register this gateway before gRPC starts accepting traffic and to renew its JWS before
+expiry. Configure `jwt_validation.datasource_url`, then enable:
+
+```yaml
+data_source:
+  enabled: true
+  auth:
+    type: "oauth_refresh"
+    client_id_env: "WEBEX_BYODS_CLIENT_ID"
+    client_secret_env: "WEBEX_BYODS_CLIENT_SECRET"
+    refresh_token_env: "WEBEX_BYODS_REFRESH_TOKEN"
+```
+
+Set the named environment variables from an authorized Service App with
+`spark-admin:datasource_read` and `spark-admin:datasource_write`. The gateway discovers an
+exact URL/schema/audience/subject match when no datasource ID is configured, reconciles
+configuration drift, and renews the token 60 minutes before expiry by default. See the
+[Configuration Reference](config/README.md#byods-datasource-lifecycle) for all settings and
+the static-token development option.
+
 ### Run
 
 ```bash
@@ -186,7 +212,7 @@ webex-byova-gateway-python/
 ├── src/
 │   ├── auth/           # gRPC JWT validation
 │   ├── connectors/     # Virtual-agent connectors
-│   ├── core/           # Gateway server, routing, and health
+│   ├── core/           # Datasource lifecycle, gateway server, routing, and health
 │   ├── generated/      # Locally generated gRPC modules
 │   ├── monitoring/     # Development monitoring interface
 │   └── utils/          # Audio utilities

config/README.md

@@ -157,6 +157,44 @@ URL. The gateway will not start with an empty value.
 See [gRPC JWT Authentication](../docs/JWT_AUTHENTICATION.md) for claims, issuers, deployment
 modes, and troubleshooting.
 
+## BYODS Datasource Lifecycle
+
+The optional `data_source` section uses `webex-byods-sdk` to discover or register the
+gateway datasource before the gRPC listener starts and renew its JWS before expiry:
+
+```yaml
+data_source:
+  enabled: true
+  fail_startup_on_error: true
+  id: ""
+  id_env: "WEBEX_BYODS_DATA_SOURCE_ID"
+  # Empty values inherit the JWT validation URL and schema.
+  url: ""
+  schema_id: ""
+  audience: "BYOVAGateway"
+  subject: "callAudioData"
+  token_lifetime_minutes: 1440
+  renewal_lead_time_minutes: 60
+  retry_interval_seconds: 60
+  auth:
+    type: "oauth_refresh"
+    client_id_env: "WEBEX_BYODS_CLIENT_ID"
+    client_secret_env: "WEBEX_BYODS_CLIENT_SECRET"
+    refresh_token_env: "WEBEX_BYODS_REFRESH_TOKEN"
+```
+
+Values ending in `_env` name environment variables; they do not contain credentials. The
+Service App needs `spark-admin:datasource_read` and `spark-admin:datasource_write`.
+
+When neither `id` nor the variable named by `id_env` supplies an ID, startup searches for a
+matching URL, schema, audience, and subject. It registers only when no match exists and
+rejects ambiguous matches. Explicit `url` and `schema_id` values must match the corresponding
+`jwt_validation` settings.
+
+For short-lived development, use `auth.type: "static"` with
+`access_token_env: "WEBEX_BYODS_ACCESS_TOKEN"`. Static access tokens cannot be refreshed;
+use OAuth refresh credentials for unattended operation.
+
 ## Logging
 
 ```yaml
@@ -222,6 +260,7 @@ Common checks:
 
 - [Local development](../docs/LOCAL_DEVELOPMENT.md)
 - [Local audio configuration](../docs/LOCAL_AUDIO_CONFIGURATION.md)
+- [BYODS datasource lifecycle](#byods-datasource-lifecycle)
 - [JWT authentication](../docs/JWT_AUTHENTICATION.md)
 - [Testing](../docs/TESTING.md)
 - [Return to the project README](../README.md)

config/config.yaml

@@ -109,6 +109,42 @@ jwt_validation:
   # Public keys are fetched from Webex identity broker and cached to improve performance
   cache_duration_minutes: 60
 
+# BYODS datasource registration and token lifecycle
+data_source:
+  # Enable to discover/register this gateway with Webex at startup and renew
+  # its JWS before expiration.
+  enabled: false
+
+  # Stop gateway startup if registration or initial renewal fails.
+  fail_startup_on_error: true
+
+  # Optional existing datasource ID. When empty, the gateway discovers a
+  # matching datasource or registers one. id_env takes precedence when id is empty.
+  id: ""
+  id_env: "WEBEX_BYODS_DATA_SOURCE_ID"
+
+  # Leave URL and schema empty to reuse the JWT validation values above.
+  # If explicitly configured, they must exactly match the JWT values.
+  url: ""
+  schema_id: ""
+  audience: "BYOVAGateway"
+  subject: "callAudioData"
+
+  # Webex supports a maximum JWS lifetime of 1440 minutes (24 hours).
+  token_lifetime_minutes: 1440
+  renewal_lead_time_minutes: 60
+  retry_interval_seconds: 60
+
+  # The SDK reads credential values from these environment variables. An OAuth
+  # refresh token supports unattended access-token renewal. A static access
+  # token can be used for short-lived development by setting type: "static"
+  # and access_token_env: "WEBEX_BYODS_ACCESS_TOKEN".
+  auth:
+    type: "oauth_refresh"
+    client_id_env: "WEBEX_BYODS_CLIENT_ID"
+    client_secret_env: "WEBEX_BYODS_CLIENT_SECRET"
+    refresh_token_env: "WEBEX_BYODS_REFRESH_TOKEN"
+
 # Logging configuration
 logging:
   gateway:

docs/CUSTOMER_EVALUATION.md

@@ -67,16 +67,25 @@ a derivative of this sample.
 3. Decide who will own the Service App, public gateway endpoint, and voice-agent connector.
 4. Create and authorize the Service App, including the Voice Virtual Agent schema and the
    gateway's data exchange domain.
-5. Make the gateway available at a public TLS-enabled server URL, then register an `ACTIVE`
-   BYOVA data source using that exact URL. Use the same value for
-   `jwt_validation.datasource_url` in the gateway configuration.
+5. Make the gateway available at a public TLS-enabled server URL and set that exact value in
+   `jwt_validation.datasource_url`. Enable automatic datasource lifecycle management to
+   discover or register the `ACTIVE` BYOVA datasource and renew its JWS, or register and
+   maintain it manually.
 6. Configure Contact Center AI and add the Virtual Agent V2 activity to a test flow.
 
 Use the current [Service App authorization steps](https://help.webex.com/default/article/5g8s6u),
 [BYODS guide](https://developer.webex.com/webex-contact-center/docs/bring-your-own-data-source-cc),
 and [BYOVA developer guide](https://developer.webex.com/webex-contact-center/docs/bring-your-own-virtual-agent)
 for the Webex onboarding flow.
 
+For the automatic path, provide the authorized Service App OAuth credentials through
+environment variables and set `data_source.enabled: true`. The gateway establishes the
+datasource before accepting gRPC traffic and prints the ID needed by the Contact Center
+virtual-agent feature. Follow
+[Local Audio Connector Configuration](LOCAL_AUDIO_CONFIGURATION.md#3-enable-automatic-byova-data-source-registration)
+for a complete sandbox example and
+[Configuration](../config/README.md#byods-datasource-lifecycle) for all lifecycle settings.
+
 For a complete first-time walkthrough, including these dependencies in the required order,
 follow the
 [BYOVA with AWS Lex setup guide](https://developer.webex.com/webex-contact-center/docs/byova-and-aws-lex).

docs/JWT_AUTHENTICATION.md

@@ -56,6 +56,39 @@ https://gateway.example.com:443
 Copy the registered value rather than reconstructing it. Use the same URL when a temporary
 development endpoint changes.
 
+### Automatic Registration and JWT Claim Alignment
+
+The optional `data_source` lifecycle can discover or register the datasource before the
+gRPC listener starts and renew its JWS before expiry. Leave its URL and schema empty to
+inherit the JWT values:
+
+```yaml
+jwt_validation:
+  enabled: true
+  enforce_validation: true
+  datasource_url: "https://your-gateway.example.com:443"
+  datasource_schema_uuid: "5397013b-7920-4ffc-807c-e8a3e0a18f43"
+
+data_source:
+  enabled: true
+  fail_startup_on_error: true
+  url: ""
+  schema_id: ""
+  auth:
+    type: "oauth_refresh"
+    client_id_env: "WEBEX_BYODS_CLIENT_ID"
+    client_secret_env: "WEBEX_BYODS_CLIENT_SECRET"
+    refresh_token_env: "WEBEX_BYODS_REFRESH_TOKEN"
+```
+
+Set the named environment variables from an authorized Service App. With empty lifecycle
+URL and schema values, registration and validation share one source of truth. If both
+sections explicitly configure either value, startup rejects a mismatch.
+
+The datasource token managed by this lifecycle is separate from each inbound request JWT:
+the lifecycle creates and renews the registration's JWS, while the interceptor validates
+the signed JWT Webex sends with a gRPC request.
+
 ### Datasource Schema UUID
 
 The standard Voice Virtual Agent schema UUID used by this sample is:
@@ -153,7 +186,8 @@ before making a key request to prevent arbitrary key-fetch URLs.
 
 ### Datasource Claims Validation Failed
 
-- Copy the exact registered datasource URL into the configuration.
+- Copy the exact registered datasource URL into the configuration, or enable automatic
+  lifecycle management so registration inherits the JWT URL and schema.
 - Check whether the registered URL includes `:443` or a trailing path.
 - Confirm the token schema UUID matches the configured Voice Virtual Agent schema.
 

docs/LOCAL_AUDIO_CONFIGURATION.md

@@ -41,7 +41,8 @@ For an end-to-end sandbox call, you also need:
 - A Webex Contact Center sandbox or organization with BYOVA enabled
 - Contact Center administrator access
 - A Webex Service App authorized by the sandbox organization
-- A BYODS data-source registration using the BYOVA schema
+- Service App credentials with `spark-admin:datasource_read` and
+  `spark-admin:datasource_write` when using automatic datasource registration
 - A publicly reachable HTTPS endpoint that supports HTTP/2 gRPC and routes to
   gateway port `50051`
 - A test entry point and a published flow containing a Virtual Agent V2 activity
@@ -207,21 +208,70 @@ The `datasource_url` must exactly match the URL used in the BYODS registration,
 including scheme, hostname, path, and any explicitly supplied port. Restart the
 gateway after changing it.
 
-### 3. Register the BYOVA data source
+### 3. Enable automatic BYOVA data-source registration
 
 Follow the Webex
 [Bring Your Own Virtual Agent](https://developer.webex.com/webex-contact-center/docs/bring-your-own-virtual-agent)
 and
 [Bring Your Own Data Source](https://developer.webex.com/webex-contact-center/docs/bring-your-own-data-source-cc)
-guides to:
+guides to confirm BYOVA is enabled, create a Service App with the required
+data-source scopes and allowed gateway domain, and have a sandbox administrator
+authorize it.
 
-1. Confirm BYOVA is enabled for the sandbox organization.
-2. Create a Service App with the required data-source scopes and add the gateway
-   hostname as an allowed domain.
-3. Have a sandbox administrator authorize the Service App.
-4. Register a data source with the BYOVA schema UUID
-   `5397013b-7920-4ffc-807c-e8a3e0a18f43` and the exact URL configured above.
-5. Save the returned data-source ID.
+Provide the authorized OAuth credentials through environment variables:
+
+```bash
+export WEBEX_BYODS_CLIENT_ID="your-client-id"
+export WEBEX_BYODS_CLIENT_SECRET="your-client-secret"
+export WEBEX_BYODS_REFRESH_TOKEN="your-refresh-token"
+```
+
+Do not commit these values. Use your deployment platform's secret manager for
+shared or long-running environments.
+
+Enable lifecycle management in `config/config.yaml`:
+
+```yaml
+data_source:
+  enabled: true
+  fail_startup_on_error: true
+  id: ""
+  id_env: "WEBEX_BYODS_DATA_SOURCE_ID"
+  # Empty values inherit jwt_validation.datasource_url and schema.
+  url: ""
+  schema_id: ""
+  audience: "BYOVAGateway"
+  subject: "callAudioData"
+  token_lifetime_minutes: 1440
+  renewal_lead_time_minutes: 60
+  retry_interval_seconds: 60
+  auth:
+    type: "oauth_refresh"
+    client_id_env: "WEBEX_BYODS_CLIENT_ID"
+    client_secret_env: "WEBEX_BYODS_CLIENT_SECRET"
+    refresh_token_env: "WEBEX_BYODS_REFRESH_TOKEN"
+```
+
+Start the gateway after the public endpoint is available. Before accepting gRPC
+traffic, it searches for an exact URL, schema, audience, and subject match. It
+reuses and reconciles one match or registers a new datasource when no match
+exists. The startup summary prints the datasource ID and JWS expiry:
+
+```text
+BYODS Datasource:
+   • Management: ENABLED
+   • ID: <data-source-id>
+   • Token Expires: <timestamp>
+```
+
+Keep the gateway running. It renews the JWS 60 minutes before expiry by default
+and retries a failed renewal every 60 seconds. If more than one existing
+datasource matches, set `WEBEX_BYODS_DATA_SOURCE_ID` to the intended ID and
+restart.
+
+For a one-time manual registration instead, leave `data_source.enabled` set to
+`false`, register the same URL and BYOVA schema through the Webex API, and save
+the returned datasource ID. Automatic renewal is disabled in that mode.
 
 No vendor credentials are needed for the local connector.
 
@@ -231,7 +281,7 @@ In Control Hub:
 
 1. Go to **Contact Center > Integrations > Features**.
 2. Create a virtual-agent feature using the authorized Service App.
-3. Use the data-source ID as the resource identifier.
+3. Use the datasource ID printed by gateway startup as the resource identifier.
 4. Give it a recognizable name such as `BYOVA Local Audio Test`.
 
 ### 5. Configure and publish the flow
@@ -297,6 +347,17 @@ JWT validation is enabled without a URL. For a localhost-only smoke test, set
 `jwt_validation.enabled` to `false`. For a sandbox call, configure the exact
 public BYODS URL and leave validation enabled.
 
+### Automatic datasource registration fails during startup
+
+- Confirm the three `WEBEX_BYODS_*` environment variables are available to the
+  gateway process.
+- Confirm the authorized Service App has both datasource read and write scopes.
+- Confirm the public URL uses a domain allowed by the Service App.
+- If the logs report multiple matches, set `WEBEX_BYODS_DATA_SOURCE_ID` to the
+  intended datasource ID.
+- Keep `fail_startup_on_error: true` for end-to-end testing so the gateway does
+  not accept traffic without a managed datasource.
+
 ### The local agent does not appear in Flow Designer
 
 - Confirm `/api/config` lists `Local Audio: Local Playback`.