"kubectl apply -f" fail with "The xxxx is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update"

Issue

When running "kubectl apply" against a Resource in Kubernetes, it fails with below error.

The <Resource> is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

Environment

Kubernetes Version: v1.17.1

Root Cause

This is because there's a resourceVersion field in last-applied-configuration annotation, which is not expected. But why there's a resourceVersion field?

We can reproduce the issue like this:

## Create a CR using "kubectl apply"
$ cat origin.yml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

$ kubectl apply -f origin.yml

## Get the yaml of the resource.
$ kubectl get memcacheds memcached-sample -o yaml > current_memcached.yml

## Modify the exported yaml, eg change size 3->4, then re-apply the modified current_memcached.yml
$ kubectl apply -f current_memcached.yml
memcached.cache.example.com/memcached-sample configured

## We can see there's a resourceVersion field in last-applied-configuration.
$ kubectl get memcacheds memcached-sample -o yaml
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"creationTimestamp":"2020-08-12T12:56:46Z","generation":5,"managedFields":[{"apiVersion":"cache.example.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:size":{}}},"manager":"kubectl","operation":"Update","time":"2020-08-24T11:18:42Z"},{"apiVersion":"cache.example.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:nodes":{}}},"manager":"manager","operation":"Update","time":"2020-08-25T00:54:37Z"}],"name":"memcached-sample","namespace":"default","resourceVersion":"302294","selfLink":"/apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/memcached-sample","uid":"5ae4fdaf-f962-4355-878c-eee75be558f3"},"spec":{"size":4},"status":{"nodes":["memcached-sample-9b765dfc8-kszwv","memcached-sample-9b765dfc8-krsvd","memcached-sample-9b765dfc8-9vxvb"]}}

## Apply the original file. (Sometimes we keep a copy of the CR before we create it, and we can use this copy to restore/modify the resource)
## It fails.
$ kubectl apply -f origin.yml            
The memcacheds "memcached-sample" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

## To fix the issue, we can remove the whole last-applied-configuration annotation in the resource
$ kubectl edit memcacheds memcached-sample

## And it should succeed this time.
$ kubectl apply -f origin.yml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
memcached.cache.example.com/memcached-sample configured

The reason behind "kubectl apply -f origin.yml" failure is that, "kubectl apply" will try patching the existing resource. It will compare the last-applied-configuration with the current yaml, and send the patch. If there a resourceVersion in last-applied-configuration, it will send the resourceVersion value, but as there's no resourceVersion in yaml, it will send a null, which cause the error.

$ kubectl -v=8 apply -f origin.yml
I0825 15:54:46.065735    1555 request.go:1068] Request Body: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"name":"memcached-sample","namespace":"default"},"spec":{"size":3}}\n"},"creationTimestamp":null,"generation":null,"managedFields":null,"resourceVersion":null,"selfLink":null,"uid":null},"spec":{"size":3},"status":null}

Resolution

- Remove the last-applied-configuration annotation in the target resource, and apply again.
- In the future, if we want to modify the configuration of a resource, modify the original yaml and apply that yaml. Or use the "kubectl edit" command against the resource. Don't export the yaml of the current resource then modify then apply.

Reference

https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/
https://github.com/kubernetes/kubernetes/issues/70674
https://blog.csdn.net/leonleow/article/details/102984129