Skip to content

Status update conflict retry discards device status entries owned by other drivers #136

Description

@thc1006

When a ResourceClaim status update hits a conflict, the retry replaces the whole status.devices list on the freshly fetched claim with a snapshot taken before the first attempt. Entries another driver wrote during the conflict window are discarded.

Where

pkg/driver/dra_hook.go, in PrepareResourceClaims:

// Store original devices list to preserve across conflict retries
originalDevices := claim.Status.Devices

err = wait.ExponentialBackoffWithContext(ctx, consts.Backoff, func(ctx context.Context) (bool, error) {
    _, updateErr := d.client.ResourceV1().ResourceClaims(claim.Namespace).UpdateStatus(ctx, claim, metav1.UpdateOptions{})
    if updateErr != nil {
        if apierrors.IsConflict(updateErr) {
            freshClaim, fetchErr := d.client.ResourceV1().ResourceClaims(claim.Namespace).Get(ctx, claim.Name, metav1.GetOptions{})
            ...
            // Copy original devices list to fresh claim
            freshClaim.Status.Devices = originalDevices
            claim = freshClaim

pkg/nri/nri.go has the same block in updateClaimNetworkDataWithRetry.

Why it matters

status.devices is shared by every driver that contributed a device to the claim, and each entry names its owner. Quoting k8s.io/api/resource/v1, AllocatedDeviceStatus:

Driver specifies the name of the DRA driver whose kubelet plugin should be invoked to process the allocation once the claim is needed on a node.

A conflict on this claim is most likely to come from exactly the other party that shares the list. Refetching and then overwriting the list with a pre-conflict copy undoes their write, and the retry then succeeds, so nothing reports it.

Suggested shape

Rebuild this driver's entries on top of the latest claim on every retry rather than restoring a whole-list snapshot: keep every entry whose Driver is not this driver, and replace or append this driver's own. AllocatedDeviceStatus identifies a device by driver, pool, device and share ID, so the merge key needs all four rather than the device name alone.

I have this implemented and can send a PR referencing this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions