Tanzu Kubernetes NotAuthenticated is set on the volume on virtualmachine - sql-server

Question

Thanks for any help on this.
I'm running a Tanzu kubernetes cluster, brand new in a dev environment. I'm trying to install MS SQL Server 2019 and am hitting a wall with this error once I apply the manifest.
The SQLserver pod fails with this:
ltkc-workers-mpqdb-556696d6f6-rhpsw
Warning FailedMount 50s kubelet, sqltkc-workers-mpqdb-556696d6f6-rhpsw Unable to attach or mount volumes: unmounted volumes=[mssql-persistent-storage], unattached volumes=[default-token-qzt5k mssql-persistent-storage]: timed out waiting for the condition
Warning FailedAttachVolume 45s (x9 over 2m53s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-697e8f96-a23b-4255-9b19-fa04aeed98ee" : rpc error: code = Internal desc = observed Error: "ServerFaultCode: NotAuthenticated" is set on the volume "fbc91ad5-b62e-4bec-8132-4f2d1c5160f0-697e8f96-a23b-4255-9b19-fa04aeed98ee" on virtualmachine "sqltkc-workers-mpqdb-556696d6f6-rhpsw"
The pv and pvc all are bound:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-697e8f96-a23b-4255-9b19-fa04aeed98ee 10Gi RWO Delete Bound default/mssql-data-claim pstore-high 67m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mssql-data-claim Bound pvc-697e8f96-a23b-4255-9b19-fa04aeed98ee 10Gi RWO pstore-high 67m
The deployment manifest is just what I downloaded from the web from various other tutorials:
apiVersion: v1
kind: Service
metadata:
name: mssql-deployment
spec:
selector:
app: mssql
ports:
- protocol: TCP
port: 1433
targetPort: 1433
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mssql-deployment
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 0
selector:
matchLabels:
app: mssql
template:
metadata:
labels:
app: mssql
spec:
terminationGracePeriodSeconds: 10
securityContext:
fsGroup: 1000
restartPolicy: Always
containers:
- name: mssql
resources:
requests:
memory: 8000Mi
image: mcr.microsoft.com/mssql/server:2019-latest
ports:
- containerPort: 1433
env:
- name: MSSQL_PID
value: "Developer"
- name: ACCEPT_EULA
value: "Y"
- name: SA_PASSWORD
value: VMware123!
volumeMounts:
- name: mssql-persistent-storage
mountPath: /var/opt/mssql
volumes:
- name: mssql-persistent-storage
persistentVolumeClaim:
claimName: mssql-data-claim
Here is the pvc yaml:
kind: PersistentVolumeClaim
metadata:
name: mssql-data-claim
spec:
accessModes:
- ReadWriteOnce
# storageClassName: vsan-default-storage-policy
storageClassName: pstore-high
resources:
requests:
storage: 10Gi
The storage class exists. I have tried this with both the default vSAN and other storage classes and always hit the same volume authentication issue.
I've searched high and low, can't find any related docs. Was hoping to see if someone knew more.
Thanks so much!!

Answer 1

Thanks again for the help, our team was able to fix this. We found out that our vCenter root password had expired. Once we reset the password our persistent volumes were able to mount to the containers without any errors. Highly suggest if you are running Tanzu to make sure your vCenter is fully updated.