Arduino + ESP8266 Mini Server with Command Que

The Problem

Up to this point, my mini web server running on my Arduino + ESP8266 is ok if things move very slow.  Once I start hitting it with several page requests and aborts, the system just crashes.  After looking through the serial logs, it became clear that I was missing a lot of incoming characters.  This was because there were several places in my old code that was blocking incoming characters while sitting and waiting for the “OK” response.  While I was waiting for the “OK”, some other client would request something and I would totally miss the request.

The (Attempted) Solution:

So I spent a lot of time (…more than I should have….) entirely re-writing my serial port handling code.  The major re-write was to add a command que that would then write the command to the serial port after some delay (points A & B under step 4 below).  The high level concept is:

  1. For each line received from the ESP8266, look for a key word (ie: “GET / “) that I need to answer.
  2. Formulate an answer string when that key word is found on an incoming line.
  3. Stick the string in a fifo style command que to be sent later.
  4. When there are no characters coming in, send the oldest command in the que out (fifo style).
    1. I had to add a time stamp check to add a delay between 2 consecutive commands being sent.  I had to do this because the ESP8266 sometimes would respond with some sort of “busy” response if I tried to send 2 commands too quickly together.
    2. I also added a time stamp check to add a delay between when I received a request to when I sent out the next response.  I just wait a while and assume that my command was successful and move on.  I am not making a commercial grade server and spent too much time here and want to move on.
  5. If a character is received from the ESP8266, go to step 1.  If no characters came from the ESP8266, keep sending commands in the que until all gone.

Best I can do in words.  You can just read the attached code below for more details.

Disclaimer: There are likely many server experts out there cringing at what I am doing.  I am no expert here and really just wanted to play around and got carried away with trying to actually serve up a web page.  Should have stopped long ago and made a custom daemon and custom client to do what I wanted.  This ended up being a long distraction… although it was fun to learn more about how servers work.

Extra Fix

HTTP header: A very helpful visitor commented in my earlier post “Mini Server on Arduino + ESP8266” that I was not sending the HTTP header.  I found very nice and simple tutorial here: http://www.jmarshall.com/easy/http/.  I added a VERY simple HTTP header to my response for HTML page and don’t have to send the connection close command any more.  The client automatically closes the port now.  It took some tricky coding but the clients seem much happier.

The Code

Tired of copy pasting the entire code.  If you want it, go here: https://drive.google.com/file/d/0B1a0nPfCQQvKX3d2OUFpSkRtQW8/view?usp=sharing

You can open it in a text editor if you don’t have the Arduino IDE.

Advertisements

Arduino Parsing for Occurrence of String from Serial Port

The Plan

For the next phase of my project, I will need to parse incoming text from the ESP8266 over a serial port and do something if I find some key words.  I started trying to dive straight in with my auto start ESP8266 server code and trying to parse responses from remote client but ended using a lot of time rebooting and starting up the client TTY session.  I got smarter and decided to just see if I can parse from the serial port from the PC.

The Implementation

Concept is simple.

  1. Read a string from the serial port
  2. Check if there is an occurrence of a certain string within the string.
  3. Do something like toggle an IO that blinks an LED.

Would be easy in C+ or Visual Basic but same commands do not exist for Arduino.  After some searching around various forums that came close to what I was looking for, I stumbled onto the function needed at the Arduino reference page.  Now that I found them, it is quite simple.  The two key finds were:

Once I found these, the code was very trivial.

The Code

#include <string.h>

void setup(){
 Serial.begin(9600);
 Serial.setTimeout(100);
 pinMode(13,OUTPUT);
 digitalWrite(13,LOW);
}

void loop(){
 String IncomingString="";
 char SingleChar;
 boolean StringReady = false;
 

 while (Serial.available()) 
 {
   IncomingString=Serial.readString();
   StringReady= true; 
 }
 if (StringReady){
   Serial.println("Received String: " + IncomingString); 
   if (IncomingString.indexOf("key") != -1) {
     Serial.println("I found 'key' in there");
     BlinkLED(13,3,300);
   } 
 }

}

void BlinkLED(int LEDPin, int NumberOfBlinks, int OnDuration)
{
 for (int x=1; x <= NumberOfBlinks ; x ++){
   digitalWrite(LEDPin,HIGH);
   delay(OnDuration);
   digitalWrite(LEDPin,LOW);
   delay(OnDuration); 
 }
}

Auto ESP8266 Server Init

The Plan

Now that I have a pretty stable HW setup, I want to start automating things and automate controls over Wifi.  The first step is to be able to auto init the ESP8266 whenever my system is power up or reset.

Implementation

The ESP8266 is able to store the last Wifi network configuration and rejoins the last Wifi access point that it was on each time it power up or is reset.  I am going to assume that the ESP8266 is set up to attach to my Wifi router for now.  Will come back and implement a way to set it up if moving to a new area later.

So assuming the ESP8266 is already on the Wifi network, I need to give the following commands….

AT+RST                // Reset the module
AT+GMR                // Just a sanity check that the ESP8266 is responding
AT+CIPMUX=1           // Set up for multi connections
AT+CIPSERVER=1,8888   // Start up the server on port 8888

For each of these commands, I want to wait for the correct response from the ESP8266.  If I don’t get the correct response to one of the commands, I should halt and somehow indicate there is a failure.  Also would be nice to indicate as each step is successful.  I could use debug prints to the serial port on the PC, but eventually I want to have the system be a stand alone module so won’t have a PC connected to watch the serial port.

To add some sort of indication, I am going to use 2 LEDs and blinking codes to indicate what is going on.  One red LED and one green LED.  Here is what the circuit looks like…

WithLEDIndicators

BTW – If you have seen the previous post, you will noticed I add a few extra features to make my life easier… LED indicator for when the 3.3V regulator is on and a switch to turn on the power to the 3.3V regulator (so I don’t have to keep unplugging the 9V battery),

The blinking code that I want to implement is…

  • Each time a command is going to be sent, the red LED will blink
  • Each time a command is successful, the green LED will blink
  • If there is a failure, both green and red LED’s will be steady on.
  • If the entire setup is good, the red and green LED will blink alternately repeatedly.
  • To allow me to know what step the code is on, the LED’s will blink the number of times equivalent to the step it is on…. 1 blink for AT+RST, 2 blinks for AT+GMR, etc….

SW Implementation

I struggled for a while to write my own AT command parser, but found a really nice one that was shared by TM here: http://hackaday.io/project/3072/instructions.  It is ingenious!! I heavily borrowed his source code and did a little tinkering to fit my needs.  Added a blinking LED functions as I described above. Here is what my source code looks like…

/*
Try to start the server autonomously
 */
#include <SoftwareSerial.h>
#include <string.h>

#define TIMEOUT 5000 // mS
#define GREENLED 4
#define REDLED 5

SoftwareSerial mySerial(7, 6); // RX, TX



void setup() 
{
 pinMode(REDLED,OUTPUT); 
 pinMode(GREENLED,OUTPUT);
 Serial.begin(9600);
 mySerial.begin(9600);

 int CommandStep = 1;
 
 //This initializes the Wifi Module as a server 
 BlinkLED(REDLED,CommandStep,300); 
 SendCommand("AT+RST", "Ready", true);
 BlinkLED(GREENLED,CommandStep,300);
 CommandStep++;
 
 BlinkLED(REDLED,CommandStep,300); 
 SendCommand("AT+GMR", "OK", true);
 BlinkLED(GREENLED,CommandStep,300);
 CommandStep++;
 
 BlinkLED(REDLED,CommandStep,300); 
 SendCommand("AT+CIPMUX=1","OK",true);
 BlinkLED(GREENLED,CommandStep,300);
 CommandStep++;
 
 BlinkLED(REDLED,CommandStep,300); 
 SendCommand("AT+CIPSERVER=1,22","OK",true);
 BlinkLED(GREENLED,CommandStep,300);
 //----------------------------------------------------


 mySerial.setTimeout(5000);
 
}

void loop() // run over and over
{
 delay(1000);
 BlinkLED(GREENLED,1,300);
 BlinkLED(REDLED,1,300);

}

void BlinkLED(int LEDPin, int NumberOfBlinks, int OnDuration)
{
 for (int x=1; x <= NumberOfBlinks ; x ++){
 digitalWrite(LEDPin,HIGH);
 delay(OnDuration);
 digitalWrite(LEDPin,LOW);
 delay(OnDuration); 
 }
}


boolean SendCommand(String cmd, String ack, boolean halt_on_fail)
{
 mySerial.println(cmd); // Send "AT+" command to module

 // Otherwise wait for ack.
 if (!echoFind(ack)) // timed out waiting for ack string
 if (halt_on_fail)
 errorHalt(cmd+" failed");// Critical failure halt.
 else
 return false; // Let the caller handle it.
 return true; // ack blank or ack found
}

// Read characters from WiFi module and echo to serial until keyword occurs or timeout.
boolean echoFind(String keyword)
{
 byte current_char = 0;
 byte keyword_length = keyword.length();
 
 // Fail if the target string has not been sent by deadline.
 long deadline = millis() + TIMEOUT;
 while(millis() < deadline)
 {
 if (mySerial.available())
 {
 char ch = mySerial.read();
 Serial.write(ch);
 if (ch == keyword[current_char])
 if (++current_char == keyword_length)
 {
 Serial.println();
 return true;
 }
 }
 }
 return false; // Timed out
}

// Print error message and loop stop.
void errorHalt(String msg)
{
 Serial.println(msg);
 Serial.println("HALT");
 digitalWrite(REDLED,HIGH);
 digitalWrite(GREENLED,HIGH);
 while(true){};
}

Testing

Not much to say other than it worked and I could see the blinking lights.  I also purposely put in a string to look for that would never occur and saw both lights turn on.  Instead of keeping to describe in words, here is a video.

ESP8266 to Arduino IO matching

The Plan

Up to this point, I have been using a R divider to interface between my 5V Arduino to my 3.3V ESP8266.  This has been working but I still fear that I am stressing the ESP8266 on it’s TX side since it is getting pulled up to 5V by the Arduino.  To address this, I bought a NTE4050B Hex Buffer/Converter.  The datasheet is here: http://www.nteinc.com/specs/4000to4099/pdf/nte4049_50B.pdf .  I would have like to have a true level shifter IC, but not easy to find at local electronics stores in the Roseville, CA area and didn’t want to wait ordering on-line.  The NTE4050B datasheet says… “These devices are intended for use as COS/MOS to DTL/TTL converters”.  This is the plan… for now.

The Setup

LevelShifterAndVariableSupply

Ardiuno Sketch

*
 Software serial test
 
 Receives from the hardware serial, sends to software serial.
 Receives from software serial, sends to hardware serial.

 */
#include <SoftwareSerial.h>

SoftwareSerial mySerial(7, 6); // RX, TX

void setup() 
{
 // Open serial communications and wait for port to open:
 Serial.begin(9600);
 mySerial.begin(9600);

 
}

void loop() // run over and over
{
 while (mySerial.available()>0 )
 Serial.write(mySerial.read());
 
 while (Serial.available()>0 ){
 mySerial.write(Serial.read());
 delay(100);
 }
}

Testing / Results

I connected everything and probed the voltage levels at the IO pins to ensure everything was working as expected…

  • Both the ESP8266 TX and RX pins are at 3.3V which was my main goal to make sure the ESP8266 did not eventually die from stress on the IO pins.
  • The Arduino SW TX pin (pin 6)  was still at 5V, but was 3.3V on the other side of the buffer which is as expected.
  • The Arduino SW RX pin (pin 7) was only at 3.3V to my surprise.  I disconnected the wire to buffer and saw that the Arduino RX pin was 5V when not connected.  This must be because there is an internal pull up resistor on that pin.  Once the buffer output is connected, it must be sinking enough current to bring it down to the 3.3V output level.  This should be ok since the Vih of the Arduino should be lower than 3.3V.

I ran the above Arduino serial pass through code and can reliably communicate with the ESP8266.

If you have been following the posts, I spent some time to find that I needed to add a 100 msec delay between characters when using R divider to do level shifting or I would lose the characters when sending more than 1 character at a time.  Just for the fun of it, I removed the 100 msec delay between characters to see if the buffer was able to clean the signal up enough.  To my surprise, the communication is still very solid and much faster.  I am wondering if I might even be able to boost up the baud rate once I start trying to do more.  Will try when needed.

I declare this addition to be a success!

Update….

One more positive side effect using this buffer.  I can now power cycle the ESP8266 without having to disconnect the IO’s from the Arduino.  Not sure exactly why, but not going to question a good thing.

Follow Up Question

I had a follow up question to provide a little more details regarding the 4050 to Arduino wiring.  The attached code has the Uno SW UART assigned to digital IO 6 and 7.

  • IO 6 is assigned to be the UART transmit (Output).
  • IO 7 is assigned to be the UART receive (Input).

The NTE4050 is connected up accordingly.  I put together a diagram to help illustrate.

SerialDataDirection

Arduino SW UART example

The Arduino Uno R3 only has 1 HW UART that is connected to the on board USB – UART chip.  I need a second one to interface with my recently purchased ESP8266 ($4 Wifi module).

  • Challenge 1: Voltage Difference
    • ESP8266 is 3.3V only IO
    • Arduino Uno R3 is 5V IO
  • Solution: Use simple R divider
    • This only is good for slow speed signals.  I plan on using only 9600 baud so this should work for now.
    • To keep is simple, I am using just 2x 10K ohm R’s on the Arduino SW UART TX pin.  The Arduino RX pin shouldn’t need anything.  Hoping that 3.3V coming from the ESP8266 is high enough for the switching level of the Arduino.
    • Here is the simple diagram of my plan…
    • FinalConnectionDiagram

Instead of jumping straight in, I am going to use an FTDI TTL-232R-3V3 cable to test the Arduino SW UART concept.  This is easier to debug since I can use TeraTerm on the PC to connect to the FTDI cable and send / receive any characters at any time to test the connects and the Arduino SW sketch.

  1. Here is the connection diagram I used…
    1.  SWUART_TestingWithFTDICable
    2. FTDI cable diagram
    3. PinoutDiagram
  2. Simple Arduino sketch for testing
    1. Concept is to pass all data from the FTDI cable connected to Arduino pins6,7 (which are NOT HW UARTs) to the Arduino HW UART.
    2. Here is my simple sketch
      1. /*
        Software serial testReceives from the hardware serial, sends to software serial.
        Receives from software serial, sends to hardware serial.*/
        #include <SoftwareSerial.h>

        SoftwareSerial mySerial(7, 6); // RX, TX

        void setup()
        {
        // Open serial communications and wait for port to open:
        Serial.begin(9600);
        mySerial.begin(9600);

        mySerial.println(“Hello, world?”);
        }

        void loop() // run over and over
        {
        if (mySerial.available())
        Serial.write(mySerial.read());
        if (Serial.available())
        mySerial.write(Serial.read());
        }

      2. It is based off of http://arduino.cc/en/Reference/softwareSerial
    3. Screen Capture of TeraTerm and Arduino
      1. TerminalScreenCapture

YAHOOO!!!!!  It worked.  Now to connect it up to the ESP8266 and see if it work.