VOLTTRON forward historian authorization failing - volttron

Question

Can someone give me a tip on how to fix a forward historian agent that is going to a BAD state installed on an edge device with an attempt to forward data to a Central instance?
This is my forward agent config on the edge device where the destination server key I got from the Central instance:
{
"destination-vip": "tcp://192.168.0.168:22916",
"destination-serverkey": "asdfasdfasdfasdfasdf"
}
On the edge device I did a vctl auth serverkey to get the edge device key and on the VOLTTRON central I did a vctl auth add to add this key of this edge device to the server.
Full traceback in the log file. When I start the agent is goes BAD in about 30 seconds, and I think the error is coming from the edge to server authentication? AND not some internal edge device agent authentication....but not sure.
2022-10-24 15:23:20,622 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core INFO: Destination serverkey not found in known hosts file, using config
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core INFO: CORE address:tcp://10.200.200.168:22916?publickey=WMdhM3co5g2keLwh5yMdFBgoxaq9g9RRFz__K6KctkI&secretkey=U6UZ9bLnbvvcDm6V5uByBbc-1tA1bUd1ZYubWgKjdG4&serverkey=GgJApMOBM8bg7HiKECP6KOwk_O8f_HslrbqPygDcoCY
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: No response to hello message after 10 seconds.
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: Type of message bus used zmq
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: A common reason for this is a conflicting VIP IDENTITY.
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: Another common reason is not having an auth entry onthe target instance.
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: Shutting down agent.
2022-10-24 15:23:20,630 (forwarderagent-5.1 2914) volttron.platform.vip.agent.core ERROR: Possible conflicting identity is: volttron1.forwarderagent-5.1_1
2022-10-24 15:23:20,829 (listeneragent-3.3 1608) __main__ INFO: Peer: pubsub, Sender: listeneragent-3.3_1:, Bus: , Topic: heartbeat/listeneragent-3.3_1, Headers: {'TimeStamp': '2022-10-24T15:23:20.826211+00:00', 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message:
'GOOD'
2022-10-24 15:23:21,219 (listeneragent-3.3 1608) __main__ INFO: Peer: pubsub, Sender: forwarderagent-5.1_1:, Bus: , Topic: heartbeat/forwarderagent-5.1_1, Headers: {'TimeStamp': '2022-10-24T15:23:21.216704+00:00', 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message:
'BAD'
2022-10-24 15:23:21,613 () volttron.platform.auth.auth_protocols.auth_zmq INFO: AUTH: After authenticate user id: 'control.connection', b'4552b404-0290-41d5-b7a8-ba4653e98e3c'
2022-10-24 15:23:21,613 () volttron.platform.auth.auth_protocols.auth_zmq INFO: authentication success: userid=b'4552b404-0290-41d5-b7a8-ba4653e98e3c' domain='vip', address='localhost:1000:1000:2943', mechanism='CURVE', credentials=['sRZnNl3PPt-_NBiyXcCtqbwG7_5dYAALmDQzXtBEFFU'], user='control.connection'
2022-10-24 15:23:21,624 (listeneragent-3.3 1608) __main__ INFO: Peer: pubsub, Sender: control.connection:, Bus: , Topic: heartbeat/control.connection, Headers: {'TimeStamp': '2022-10-24T15:23:21.623967+00:00', 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message:
'GOOD'
2022-10-24 15:23:21,631 (forwarderagent-5.1 2914) volttron.platform.vip.agent.subsystems.auth INFO: Skipping updating rpc auth capabilities for agent volttron1.forwarderagent-5.1_1 connecting to remote address: tcp://10.200.200.168:22916?publickey=WMdhM3co5g2keLwh5yMdFBgoxaq9g9RRFz__K6KctkI&secretkey=U6UZ9bLnbvvcDm6V5uByBbc-1tA1bUd1ZYubWgKjdG4&serverkey=GgJApMOBM8bg7HiKECP6KOwk_O8f_HslrbqPygDcoCY
2022-10-24 15:23:21,631 (forwarderagent-5.1 2914) volttron.platform.vip.agent.subsystems.auth WARNING: Auth entry not found for volttron1.forwarderagent-5.1_1: rpc_method_authorizations not updated. If this agent does have an auth entry, verify that the 'identity' field has been included in the auth entry. This should be set to the identity of the agent
2022-10-24 15:23:30,627 (forwarderagent-5.1 2914) __main__ ERROR: Couldn't connect to address. gevent timeout: (tcp://10.200.200.168:22916)
2022-10-24 15:23:30,628 (forwarderagent-5.1 2914) __main__ ERROR: Could not connect to targeted historian dest_vip tcp://10.200.200.168:22916 dest_address None
2022-10-24 15:23:30,755 () volttron.platform.vip.agent.core INFO: CORE address:inproc://vip
2022-10-24 15:23:30,786 () volttron.platform.vip.agent.core INFO: Connected to platform: router: 4552b404-0290-41d5-b7a8-ba4653e98e3c version: 1.0 identity
One thing to note is I am trying to rectify a botched update from 8.0 to 8.2 on the edge device I ended up just recloning the repo and bootstrapping 8.2, so now I have this volttron1 config someone.

Answer 1

#bbartling, on the destination Volttron platform, can you verify that all remote connections (e.g. Forwarder connections) have been received and approved? You can run the following commands on the destination platform to verify:
# to view all remote connections; this will return a list of user ids that can be used to approve or deny connections
vctl auth remote list
# to approve a remote connection
vctl auth remote approve <USER_ID>
Docs to vctl auth remote commands
If you're goal is to verify that your forwarder can connect to the destination platform, you can always auto accept all remote connections (though you should only do this for testing since it is not secure):
vctl auth add --credentials '/.*/'

Answer 2

I fixed this by doing:
vctl auth serverkey on the central instance and adding this into
the forward agent config file on the edge device
Then on the edge device doing a vctl auth publickey which flashes the
edge device key of the forward agent which then needs to be added to the central instance
on a vctl auth add