ESP32 arduino x509 format is invalid loading from SPIFFS - aws-iot

Question

I'm trying to fetch 3 keys from SPIFFS that I stored from a string to a file (when fetched from an API endpoint).
I am able to read the files using the SPIFFS library and print the values out in the serial console. When I check the contents, they are just fine. When I use the contents in the secureClient.setCACert() function, they "break" and throw an exception; E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():138]: (-8576) X509 - The CRT/CRL/CSR format is invalid, e.g. different type expected
My code looks like the following:
#include <WiFiClientSecure.h>
#include <FS.h>
#include <SPIFFS.h>
WiFiClientSecure net = WiFiClientSecure();
String read2String(const char * path) {
String output = "";
File file = SPIFFS.open(path, "r");
if (!file || file.isDirectory()) {
return "";
}
while (file.available()) {
char c = file.read();
output.concat(c);
}
file.close();
return output;
}
void readFilesToVariables() {
String awsRootCa = read2String("/AmazonRootCA1.pem");
String privateKey = read2String("/private.pem.key");
String certificate = read2String("/certificate.pem.crt");
net.setCACert(awsRootCa.c_str());
net.setCertificate(certificate.c_str());
net.setPrivateKey(privateKey.c_str());
}
void setup() {
Serial.begin(115200);
SPIFFS.begin();
readFilesToVariables();
mqttClient.begin(AWS_IOT_ENDPOINT, AWS_IOT_ENDPOINT_PORT, net); // crashes here (I think)
...
}
I have on the mqttClient.begin(...) line (the mqtt client and all the variables except for the net var) a couple of variables that are defined in my script but not shared here.
Anyone any idea what I'm doing wrong here?
the exact same code works when I define the certificates and private key using the following methodology:
static const char AWS_CERT_CA[] PROGMEM = R"EOF(
-----BEGIN CERTIFICATE-----
certificate_content_here
-----END CERTIFICATE-----
)EOF";
What am I doing wrong?

Answer 1

The same thing drove me crazy for the last few days.
Then I found a solution, but only supposed the problem.
I think that this issue has something to do with memory handling and where and how you store the certificates after you read it from flash.
Here's how I read the certificates. Using calloc I allocate the memory and never free it, so that the certificate will always be available to the MQTT client
char* getFileAsString(String path) {
File file = SPIFFS.open(path);
if (!file) {
Serial.println("Failed to open " + path);
return NULL;
}
char* buffer = (char*)calloc(file.size(), sizeof(char));
file.readBytes(buffer, file.size());
file.close();
return buffer;
}
Then I just use it in the config struct
const esp_mqtt_client_config_t mqttConf = {
.uri = MQTT_BROKER,
.client_id = MQTT_CLIENT_ID,
.cert_pem = getFileAsString("/caFile"),
.client_cert_pem = getFileAsString("/certificateFile"),
.client_key_pem = getFileAsString("/privateFile"),
};
Note: I did't put the *_len attributes, because I stored my certificates with a leadinig null character:
dd if=/dev/zero bs=1 count=1 >> [my cert name].pem
You can use regular certificates, but you must set the certificates length or adding 1 byte to the calloc instruction (calloc will set all the allocated memory to 0x00, so there will be a leading null character):
char* buffer = (char*)calloc(file.size() + 1, sizeof(char));