You need to log in to create posts and topics.

Adjustments for Binary Payloads with ESP01, Arduino, and DHT22

NeoDarwin
NeoDarwin@neodarwin
4 Posts
#1 · January 10, 2025, 6:59 pm

While working with the ESP01 module in combination with an Arduino board and a DHT22 sensor, I noticed a limitation: the payload generated was in string format. This posed a significant challenge for implementing the MQTT protocol, which often relies on more efficient binary payloads, especially for constrained devices and networks.

To address this issue, I made some adjustments to allow binary payloads to be used effectively. This change enables the payload to handle raw binary data, making it more suitable for MQTT implementations. These adjustments improve the compatibility and efficiency of the communication, particularly when dealing with data transmission in IoT projects.

With this update, we can appyl the MQTT Protocol and we can catch the different payload with MQTT (over TCP/IP)


We can connect our smartphone with a MQTT software to pick up the data from the simuleIDE 

Suggestion to improve the esp01 code :

Original Source

The original source code for the ESP01 module focused on basic AT command handling and TCP data transfer. Key features include:

  1. Basic Command Handling: The command() method processes incoming AT commands and responds with predefined replies like OK or ERROR.
  2. TCP Data Management: Functions like tcpSend() and tcpConnected() handled basic TCP operations. However, they primarily worked with raw strings, making them unsuitable for complex protocols like MQTT that rely on binary payloads.
  3. Debugging Support: Debug messages were included to track actions such as sending data, connecting, and disconnecting from hosts.
  4. Limitations:
    • No support for MQTT protocol handling.
    • Payload processing was limited to strings, making it difficult to work with binary data.
    • The TCP connection logic didn't validate the existing connection state effectively, leading to potential redundant operations.

Updated Source

The updated source introduces significant improvements to support MQTT binary payloads and optimize TCP operations. Key changes include:

  1. MQTT Binary Payload Handling:

    • New Method: handleMQTTFrame() was added to parse MQTT frames, extract topics, and payloads, and handle specific MQTT packet types like CONNECT, PUBLISH, SUBSCRIBE, and PINGREQ.
    • Payload as Binary Data: The byteReceived() method now processes binary data instead of relying solely on string-based commands. This allows for more versatile and efficient data handling.

  2. TCP Enhancements:

    • Added tcpSocket->flush() in tcpSend() to ensure immediate transmission of data.
    • Improved connectTcp() logic to validate the current connection state and avoid unnecessary reconnections.
    • Enhanced logging to provide more clarity during connection and data transfer operations.

  3. Streamlined Command Parsing:

    • The command buffer now supports both AT commands and binary payloads. Binary data is handled via the MQTT-specific logic, while AT commands are processed as before.
    • Additional checks ensure proper handling of incomplete or invalid frames.

  4. New MQTT Packet Support:

    • CONNECT: Detects and processes the initial MQTT connection request.
    • PUBLISH: Extracts and logs the topic and payload from the publish packet.
    • PINGREQ and DISCONNECT: Handles these MQTT-specific control packets appropriately.

  5. Debugging and Logging:

    • More detailed debug logs for MQTT frames, topics, and payloads.
    • Logs indicate unsupported MQTT packet types for easier troubleshooting.

Example Improvements

  • Original TCP Send Logic:

    • Did not explicitly flush data.
    • Limited to string-based payloads.

int bytes = tcpSocket->write(m_tcpData);

  • Updated TCP Send Logic:

    • Flushes data to ensure immediate transmission.
    • Works with binary payloads.
 int bytes = tcpSocket->write(m_tcpData); tcpSocket->flush(); // Force immediate data transmission
  • MQTT Frame Handling:
    • The new method handleMQTTFrame() parses and handles MQTT-specific frames. For example:
    • For a CONNECT frame:
if (packetType == 1) {
    if (m_debug) qDebug() << "Esp01 - MQTT CONNECT Frame Received";
    m_action = tcpSend;
    updateStep();
}
  • Optimized Connection Management:
    • Verifies the connection state before attempting to reconnect
if (tcpSocket->state() == QAbstractSocket::ConnectedState &&
    tcpSocket->peerName() == m_host &&
    tcpSocket->peerPort() == m_port) {
    if (m_debug) qDebug() << "Esp01 - Socket already connected to" << m_host << m_port;
    return;
}

Benefits of the Updated Source

  1. MQTT Support: The updated code is now capable of parsing and handling MQTT binary frames, making it suitable for IoT applications.
  2. Improved Efficiency: Optimized TCP connection logic avoids unnecessary reconnections and ensures faster operations.
  3. Enhanced Debugging: More detailed logs help in understanding the behavior of the ESP01 module during development and troubleshooting.
  4. Binary Data Compatibility: The ability to process binary payloads expands the module's versatility.

Conclusion

The updated source significantly improves the ESP01 module by enabling binary payload handling and supporting MQTT-specific operations. These changes make it more suitable for modern IoT applications where protocols like MQTT are widely used.

We need to update the esp01.h

To test the MQTT Protocol, I created a Raspberry VM on Virtual Box and I installed the mosquitto solution, I created the sketch to publish the temperatur below the mqtt protocol.

Arduino sketch to test it

#include <SoftwareSerial.h>
#include <DHT.h>

// Definir les broches et le type de capteur DHT
#define DHTPIN 4
#define DHTTYPE DHT22
DHT dht(DHTPIN, DHTTYPE);

// Definir les broches RX et TX pour l'ESP-01
SoftwareSerial espSerial(2, 3); // RX, TX

// Topics MQTT
const char* topic_temp = "BTS-CIEL/A125/temperature";

// Variables de timing
unsigned long lastMsg = 0;
unsigned long lastPing = 0;
#define INTERVAL 5000    // Intervalle de publication (5 secondes)
#define KEEP_ALIVE 60000 // Intervalle de ping MQTT (60 secondes)

void setup() {
  // Initialisation des ports serie
  Serial.begin(9600);
  espSerial.begin(115200);
  
  // Initialisation du capteur DHT
  dht.begin();

  // Connexion au WiFi et au broker MQTT
  connectWiFi();
  connectMQTT();
}

void loop() {
  unsigned long now = millis();

  // Envoyer un PINGREQ toutes les 30 secondes
  if (now - lastPing > KEEP_ALIVE) {
    lastPing = now;
    sendPing();
  }

  // Lire la temperature et publier MQTT toutes les 5 secondes
  if (now - lastMsg > INTERVAL) {
    lastMsg = now;
     // DisconnectReq();
    // Verifiez si la connexion au broker MQTT est toujours active
    if (!checkConnection()) {
      connectMQTT(); // Reconnectez si necessaire
    }
   
    // Lire les donnees du capteur
    
    float temperature = dht.readTemperature();
    if (isnan(temperature)) {
      Serial.println("Erreur de lecture du DHT22");
      return;
    }

    // Publier les donnees sur le topic MQTT
    
    publishMQTT(topic_temp, temperature);

    Serial.print("Temperature: ");
    Serial.print(temperature);
    Serial.println(" °C");
  
  }
}

void connectWiFi() {
  sendCommand("AT+RST", "OK");
  delay(2000);
  sendCommand("AT+CWMODE=1", "OK");
  sendCommand("AT+CWJAP=\"VotreSSID\",\"VotreMotDePasse\"", "OK");
}

void connectMQTT() {
  sendCommand("AT+CIPMUX=0", "OK");
  sendCommand("AT+CIPSTART=\"TCP\",\"127.0.0.1\",1883", "OK");
  delay(1000);

  // Paquet CONNECT avec Keep Alive = 60s
  byte connectCmd[] = {
    0x10, 0x1A,                   // Fixed Header
    0x00, 0x04, 'M', 'Q', 'T', 'T', // Protocol Name
    0x04,                         // Protocol Level
    0x02,                         // Flags (Clean Session)
    0x00, 0x3C,                   // Keep Alive (60s)
    0x00, 0x0E,                   // Client ID Length
    'M', 'Q', 'T', 'T', 'A', 'r', 'd', 'u', 'i', 'n', 'o','U', 'n', 'o' // Client ID
  };

  sendCommand("AT+CIPSEND=" + String(sizeof(connectCmd)), ">");
  espSerial.write(connectCmd, sizeof(connectCmd));
  Serial.println("CONNECT command sent");

  // Attendre la reponse du broker
  // Nous attendons le connect Act
  delay(1000);
String response = "";

// Lire les donnees du port serie
while (espSerial.available()) {
  response += char(espSerial.read()); // Lire caractere par caractere
}

// Verifier la connexion MQTT
bool mqttSuccess = response.indexOf((char)0x20) != -1 && response.indexOf((char)0x00) != -1;

if (mqttSuccess) {
  Serial.println("Connexion MQTT etablie avec succes !");
} else {
  Serial.println("Erreur : Connexion MQTT echouee.");
  Serial.print("Reponse brute du broker (en hexadecimal) : ");
  
  // Afficher la reponse en hexadecimal pour faciliter le debugging
  for (size_t i = 0; i < response.length(); i++) {
    Serial.print("0x");
    if (response[i] < 16) Serial.print("0"); // Ajouter un 0 pour les nombres < 16
    Serial.print((int)response[i], HEX);
    Serial.print(" ");
  }
  Serial.println();
  Serial.println(response);
}

}
void publishMQTT(const char* topic, float value) {
    // Convertir la valeur en une chaîne avec 2 décimales
    String valueStr = String(value, 2);
    
    // Calculer les longueurs du topic et du payload
    size_t topicLen = strlen(topic);
    size_t payloadLen = topicLen + 2 + valueStr.length(); // 2 octets pour la longueur du topic

    // Vérifiez les limites du protocole MQTT
    if (topicLen == 0 || payloadLen > 127) {
        Serial.println("Erreur : Longueur invalide du topic ou payload");
        return;
    }

    // Construire le paquet MQTT
    byte publishCmd[2 + payloadLen];
    publishCmd[0] = 0x30;                      // Fixed Header (type PUBLISH, QoS 0)
    publishCmd[1] = payloadLen;               // Longueur du paquet (Variable Header + Payload)
    publishCmd[2] = (topicLen >> 8) & 0xFF;   // MSB de la longueur du topic
    publishCmd[3] = topicLen & 0xFF;          // LSB de la longueur du topic
    memcpy(&publishCmd[4], topic, topicLen);  // Copie du topic
    memcpy(&publishCmd[4 + topicLen], valueStr.c_str(), valueStr.length()); // Copie du payload (valeur)

    // Envoyer la commande AT pour transmettre les données
    String cmd = "AT+CIPSEND=" + String(2 + payloadLen);
    sendCommand(cmd, ">");
    espSerial.write(publishCmd, sizeof(publishCmd));
    delay(1000);

    // Debug : Afficher le message publié
    Serial.println("Message publié sur le topic: " + String(topic) + " avec la valeur: " + valueStr);
}


void sendPing() {
  byte pingCmd[] = {0xC0, 0x00}; // MQTT PINGREQ
  sendCommand("AT+CIPSEND=2", ">");
  espSerial.write(pingCmd, sizeof(pingCmd));
  delay(1000);
  Serial.println("PINGREQ envoye au broker MQTT");
}

void DisconnectReq() {
  byte pingCmd[] = {0xE0, 0x00}; // MQTT DISCONNECT REQ
  sendCommand("AT+CIPSEND=2", ">");
  espSerial.write(pingCmd, sizeof(pingCmd));
  delay(1000);
  Serial.println("DISCONNECT REQ envoye au broker MQTT");
}

bool checkConnection() {
  sendCommand("AT+CIPSTATUS", "OK");
  String response = "";
  while (espSerial.available()) {
    response += espSerial.readString();
  }

  // STATUS:3 signifie que le socket est connecte
  if (response.indexOf("STATUS:3") != -1) {
    return true;
  } else {
    Serial.println("Connexion au broker perdue, reconnexion...");
    return false;
  }
}

void sendCommand(String command, String expectedResponse) {
  espSerial.println(command);
  delay(1000);
  String response = "";
  while (espSerial.available()) {
    response += espSerial.readString();
  }
  if (response.indexOf(expectedResponse) == -1) {
    Serial.println("Erreur: " + command);
    Serial.println("Reponse reçue: " + response);
  }
}

 

I hope this post is helpful to you.

Click for thumbs down.0Click for thumbs up.0
Last edited on January 10, 2025, 7:02 pm by NeoDarwin
arcachofo
arcachofo@arcachofo
651 Posts
#2 · January 10, 2025, 8:16 pm

Thank you very much, I will have a look.

Click for thumbs down.0Click for thumbs up.1
NeoDarwin has reacted to this post.
NeoDarwin