ChipChop Support Forum
Log in
Log out
Join the forum
My Details
REPLIES: 17
VIEWS: 154
BACK
REPLY TO THIS POST
TomaszB
02 Nov 2024
ChipChop memory consumption (ESP8266)
Hi

I observe that ChipChop uses quite significant of heap memory. Below, there is a very short test code based on demo from ChipChop page. It seems that starting ChipChop caused allocation of almost 60% of ESP8266 available heap memory (46968-18256) / 48184. In case of this hardware it is very painful.
Do you observe similar behaviour?

Kind regards
Tomasz


Logged parameters:
Initial MEMORY Free Heap Size=48184
WiFi Connecting...
Memory BEFORE ChipChop.start Free Heap Size=46968
Memory AFTER ChipChop.start Free Heap Size=18608
0.00
Memory AFTER updateStatus Free Heap Size=18256
1.00
Memory AFTER updateStatus Free Heap Size=18256
2.00
Memory AFTER updateStatus Free Heap Size=18608





void setup(){
Serial.begin(9600);
delay(1000);
Serial.println("Initial MEMORY Free Heap Size="+String(ESP.getFreeHeap()));
WiFi.begin(SSID, PASS);

Serial.println("WiFi Connecting...");
int connRes = WiFi.waitForConnectResult(10000); //wait up to 5 seconds to establish the connection

if(connRes != WL_CONNECTED){
Serial.println("Cannot connect to the WiFi");
}

Serial.println("Memory BEFORE ChipChop.start Free Heap Size="+String(ESP.getFreeHeap()));
ChipChop.start(server_uri, uuid, device_id, auth_code);
Serial.println("Memory AFTER ChipChop.start Free Heap Size="+String(ESP.getFreeHeap()));

test=0;
ticker.attach_scheduled(10,[](){
ChipChop.updateStatus("test_numeric",test);
ChipChop.updateStatus("uptime",String(100+test));
Serial.println(test);
test++;
Serial.println("Memory AFTER updateStatus Free Heap Size="+String(ESP.getFreeHeap()));

});
}

void loop(){
ChipChop.run();
}









Gizmo
03 Nov 2024

lol... man, that's a lot of pain in your message! 🥲

Most of my house still runs on esp8266 D1-mini (although I am slowly migrating to esp32 C3 supermnini) and I never had issues with the heap but that's because I don't use esp8266 for very complex setups where you have a ton of sensors etc. I love the esp8266 but I know their limitations.

ok, so....unfortunately you can't blame ChipChop for the heap size, the library is not that big and I've tried making cut down versions stripping all logging messages, time handling etc and the most I could shave of is about 1.5kb of the heap.

The culprit is the encryption and that's handled by a lower level code which is out of my control, if you want encrypted communication then it will cost you...a lot !

Here's the proof:

1. change in the String server_uri = "wss://api1.chipchop.io/wsdev/ the wss:// to ws:// (remove the second letter 's' ) >> ws://api1.chipchop.io/wsdev/

2. use the same test code and you can also add in the loop()




int test = 0;
unsigned long loopCycle = 0;
void loop(){

    ChipChop.run();

    if ((millis() - loopCycle) > 10000){

        Serial.println("Memory BEFORE updateStatus Free Heap Size="+String(ESP.getFreeHeap()));

        ChipChop.updateStatus("test_numeric",test);
        ChipChop.updateStatus("uptime",String(100 + test));
        test++;

        Serial.println("Memory AFTER updateStatus Free Heap Size="+String(ESP.getFreeHeap()));

        loopCycle = millis();
    }

    
}



Let me know the results 😉

G

p.s. If you want I can explain in more detail why you will most likely have a big difference between wss:// and ws://


FreeWilly
03 Nov 2024

let me guess >> WiFiClientSecure.h 💣

@Gizmo, when you say ws:// instead of wss:// is that then using wifiClient.setInsecure()?
TomaszB
03 Nov 2024

Hi

1) Following your suggestion, I've added Heap info before and after ChipChop.updateStatus(). Values are exactly the same:
Memory AFTER updateStatus Free Heap Size=18712
Memory BEFORE updateStatus Free Heap Size=18712
8.00
Memory AFTER updateStatus Free Heap Size=18712
Memory BEFORE updateStatus Free Heap Size=18712
9.00
Memory AFTER updateStatus Free Heap Size=18712
Memory BEFORE updateStatus Free Heap Size=18712
10.00
Memory AFTER updateStatus Free Heap Size=18712
Memory BEFORE updateStatus Free Heap Size=18712
11.00
Memory AFTER updateStatus Free Heap Size=18712
Memory BEFORE updateStatus Free Heap Size=18712

It means that memory has been allocated but has not been freed.



ticker.attach_scheduled(10,[](){
Serial.println("Memory BEFORE updateStatus Free Heap Size="+String(ESP.getFreeHeap()));
ChipChop.updateStatus("test_numeric",test);
ChipChop.updateStatus("uptime",String(100+test));
Serial.println(test);
test++;
Serial.println("Memory AFTER updateStatus Free Heap Size="+String(ESP.getFreeHeap()));

});


2) I've tried to modify connection string to ws instead of wss: String server_uri = "ws://api1.chipchop.io/wsdev/". Yes, in this case it doesn't devour memory.

Initial MEMORY Free Heap Size=48152
WiFi Connecting...
Memory BEFORE ChipChop.start Free Heap Size=46936
Memory AFTER ChipChop.start Free Heap Size=46576
Memory BEFORE updateStatus Free Heap Size=45928
0.00
Memory AFTER updateStatus Free Heap Size=45904
Memory BEFORE updateStatus Free Heap Size=46000
1.00
Memory AFTER updateStatus Free Heap Size=46000
Memory BEFORE updateStatus Free Heap Size=46000
2.00
Memory AFTER updateStatus Free Heap Size=46000

3) I've modified your library a bit to ensure that strings are stored in PROGMEM using F macro, like F("ChipChop onEventsCallback opening connection"), however, the improvement was negligible (so i examples above, I used an original one).

Questions:
1) Why does encryption of a such short message use so much memory?
2) Why this memory is not freed after sending a message?
3) The ugliest workaround would be: switch off / suspend ChipChop for a moment in a way that it releases allocated memory THEN perform some operations that require more heap memory and THEN resume / switch ChipChop on. Can you suggest what commands shall be used with ChipChop?
Gizmo
03 Nov 2024

As @FreeWilly guessed it's the WiFiClientSecure.h library that handles the ssl encryption and that's what inflates the heap.

It's all part of the arduino-espressif that get's compiled with all other code and I wouldn't know where to start as there is about 39 files just handling wifi and http.

I don't know why it all takes so much memory, but I know that it must need it.

If you think about it:

- the data arrives encrypted so it can't be used straight away, you need the complete message to arrive (packet by packet) before you can decrypt it.
- The data packets need to be stored somewhere temporarily and it has to be the heap as the data packets don't arrive in order synchronously and you need the stack for the program to keep running
- So, you are already having one copy of the data in the heap, then you start the process of decrypting so again you are having another copy before it can be used.
- you also need to keep the ssl public key that you receive from the server for the encryption (that alone is probably 2kb) and whatever other server session parameters etc

I think probably either most of that heap space is permanently full of necessary crap needed for ssl or the heap gets too fragmented after the first request that it just stays in that state.

If you want to experiment with stopping ChipChop have a look in the library, there is a ChipChopManager::restartSockets(),
you could split that into two public methods like ChipChop.kill() and ChipChop.reload() and in the kill() destroy the websocket and see if it will destroy the WiFiClientSecure and release the heap and then restart the sockets and see what happens.
Also, be aware, that if you destroy the ChipChop connection you will most likely have to wait possibly up to 10-15 seconds to re-establish the connection which voids the whole idea of real-time bi-directional communication.

WARNING: The "Connection Watch" that runs on the ChipChop server may decide to ban your device if it detects too many connect/disconnects and decides that it's an attack or an unstable connection. I've made it so it's out of my control so do it at your own risk or for educational reasons but not as a permanent solution.


Anyway, my attitude is with the heap, if it's reaches a point where it's saturated but fully stable and doesn't move I don't touch it and live with it but at least I know pretty accurately how much "real" empty heap space I have.

I really don't know what else to say, the ssl on the esp8266 is definitely not the best and I don't know if it could be made any better on the esp8266. Let me know if you manage to achieve some improvements but I think Espressif is probably looking to phase out the esp8266 anyway in the near future and the esp32-c3 is I think the alternative replacement, with the C3 there are no issues with the available space and speed is double.

Keep me posted anyway if you come up with some improvements

G
TomaszB
04 Nov 2024

Thanks Gizmo for instant answers and useful hints. Much appreciated!

1. I need to have more heap memory only during device setup, where I ran there also WebServer to collect logs (as I use ESP serial to control another device) and adjust some parameters. So, deactivation of ChipChop will be not in a continuous loop, but rather very rare, when configuration has to be changed.
2. Thanks for hint with modification / split of restartSockets method. It is written in a very clear way. Will try to do that.

I can imagine that encryption / decryption process drains memory and understand that we can't do anything with it, but still can't comprehend how they landed Apollo vehicle on moon having 32kB of RAM....

Kind regards
Tomasz
Gizmo
04 Nov 2024

haaaaaaa, yes, 32kb moon landing! 🙈

Actually, the first PC I’ve used professionally to make a living had a 26mhz processor, I think 2mb ram and maybe 100mb hard drive and I was doing a ton of commercial work on it:-) technically your little esp8266 is 3x faster so yeah, something must have gone wrong how we write software today!

Anyway, I’ve actually done a very quick and dirty test killing and reloading the poor little ChipChop library on an esp8266 and I can confirm that destroying the wificlientsecure.h does free instantly all of the heap (goes from 18 to 45 straight away)
You have to be carful though how you do it and the order of things as I’ve crashed my esp multiple times but it can definitely be done for what you need.

I am happy to assist you with the code if you want, if it’s not something super confidential you can post it here or email me directly and I’ll revise it.

TomaszB
05 Nov 2024

Hi @Gizmo

Following your advice I have modified your library.
1) When my module needs memory (to run WebServer) it calls ChipChop.killSockets() and then stops calling ChipChop.run() in the loop.
2) When my module stops WebServer it calls ChipChop.reloadSockets() and then starts calling ChipChop.run() in the loop.

So far I observe crash only once and I suspect that WebServer hasn't released all allocated memory so ChipChop had no space to reloadSockets. But I have some idea how to improve my code a bit.

BTW - I use a bit different logging thus I've replaced all log calls with my own.

Question: Is the logic presented above fine from ChipChop stability perspective?



void ChipChopManager::killSockets(){
if(_LOG_ENABLED == true) Logger::addToLog(Logger::LogLevel::info,F("ChipChopManager::restartSockets Destroying WebSockets"));
//log("Destroying WebSockets");
webSocket->close();
connected = 0;
handshakeComplete = 0;

// Serial.print("current heap size: ");
// Serial.println(ESP.getFreeHeap());
delete webSocket;
webSocket = NULL;
}
void ChipChopManager::reloadSockets(){
webSocket = new WebsocketsClient();

webSocket->onMessage(onMessageCallback);
webSocket->onEvent(onEventsCallback);
_last_chipchop_connection_ok = millis();

if(_LOG_ENABLED == true) Logger::addToLog(Logger::LogLevel::info,F("ChipChopManager::restartSockets Restarting WebSockets"));
//log("Restarting WebSockets");
_connect_socket();
}

void ChipChopManager::restartSockets(){
killSockets();
reloadSockets();
}



Tomasz
Gizmo
05 Nov 2024

Hi Mr T,

I am such an idiot! You are using the ChipChop library vs 1.40, right?

You just need to use the Code Builder in the Dev Console to get the newer lib vs 1.4x, that one has a built in ChipChop.pause() and ChipChop.unPause(), it doesn't destroy the socket fully but it should get rid of the ssl crap and also sends a proper "disconnect" signal to the server so the re-connect later on is a lot faster.

You can also keep the ChipChop.run() no problems, it won't affect anything

I've just checked and it definitely frees the heap as soon as you call .pause()

1. In the Dev Console go in the Code Builder and pick any of your devices and any board.
2. don't need to specifiy any plugins, just click "next"
3. select the ChipChop engine and press "save" and then "next"
4. download the version you want (Platformio or Arduino Ide)

You will find in the downloaded zip everything that you need, the only files you need are
    - ChipChopEngine.h & cpp
    - ChipChop_Config.h

and in the main.cpp /.ino just replace the previous #include ChipChopManager with:


    // you may have to use quotes in the path " " instead of < > if you are using Arduino IDE
    #include <ChipChop_Config.h>
    #include <ChipChopEngine.h>
    extern ChipChopEngine ChipChop;



Alternatively, what you've done already is also ok, if you are going to use the "kill" method then just make sure that whilst the sockets are dead that you don't accidentally call ChipChop.updateStatus() or triggerEvent() as that would crash it.
If you go for the new lib then that's not a problem.

p.s. If you had a problem with your webserver releasing the memory, I guess you can just keep checking the heap size until there is around 30kb available for ssl and then re-load the sockets?


Let me know how it goes



TomaszB
07 Nov 2024

Hi @Gizmo

Let me perform test with the newest library version, I need a day or two as I have been swamped by other urgent tasks.

BTW. Yes indeed, I was calling updateStatus() after killSockets(), but I observe crashes only AFTER resume attempt (reloadSockets()).

Will let you know about results and once again thanks for help!

Tomasz
TomaszB
14 Nov 2024

Hi @Gizmo

So far it looks good and stable (I haven't noticed any memory leaks). Nevertheless, I plan to reboot every time when I finish changing config, thus pausing / resuming ChipChop should not happen too often.

Next topic: I would like not to connect permanently to Wifi, but rather do it every hour for one minute to update ChipChop status and then disconnect. So far it seems to work without any ChipChop pausing. It handles such interrupted connectivity.

Question: Shall I pause ChipChop (or call any other method) before WiFi disconnection, after reconnection to eliminate any unexpected behaviour / memory leak?

Kind regards
Tomek
Gizmo
14 Nov 2024

Hi Mr T

Yes, you can do that no problems, you have couple of options:

1. If you intend to restart the ESP then you can just call ChipChop.closeConnection() before you restart (maybe give it 100ms delay before restart)

This will send a proper "connection close" packet to your ChipChop API server so it can cleanly release the connection from memory and after restart the next re-connect will be pretty much instant without delay.

2. If you are closing the WiFi then I would say probably call ChipChop.pause() and then next time the connection is established call ChipChop.unPause()

The difference is that closeConnection() sends the "connection close" packet and just cuts the connection temporarily, it's used internally by the library to make clean re-connects and there is a timer (lastReconnectTime) running every 15 seconds that will re-attempt a connection

The pause() on the other hand calls closeConnection() and also raises a _CAN_RUN flag that completely stops any connection attempts, heartbeats etc until unPause() is called (this also reduces the workload during the paused stage)
So, pause() I think is probably better suited for longer periods of inactivity and once you restore the WiFi just call unPause() and keep going.

The only thing you shouldn't use in your scenario is the Keep Alive plugin (you probably are not anyway), that bugger keeps an eye on the actual WiFi & sockets and will restart the ESP on it's own after 2 minutes of a dead connection :-)


Question, that setup pretty much makes your device just a one way reader are you killing the WiFi because you are running your thingy on batteries or it's a security concern?



TomaszB
14 Nov 2024

Hi Gizmo

Thanks for the reply. As usual it was instant!

Yes, you are right, in this project I need only to send some data to the ChipChop cloud and I don't need any actions. Then cloud does everything what I need: sends notification where some parameters are out of allowed values or when connectivity with IoT is lost longer than a few hours.

But your comment about keepalive plugin has (unfortunately) inspired me. Can I somehow get a callback from ChipChop when a connection with backend is successfully established? I would like to add functionality that more less covers keepalive but with longer timescale (in case when ChipChop backend can't be contacted more than 3 hours the esp would be restarted).

Regards
Tomasz
Gizmo
14 Nov 2024

lol...that's the spirit!

what you need I have already done with Keep Alive and it can be set to whatever timeout you want (or whatever is the largest millis() integer you can specify on the esp)

I'll give you a bunch of info but I can tell you, writing the Keep Alive gave me a ton of headaches for such a small bit of code so if you can use it as it is do it and save yourself few mental aneurysms! :-)

Firstly, in the code that you get from Code Builder I've implemented a sort of plugin structure and most plugins have builtin event emitters that you can hook into and get informed when relevant things happen.

You can see roughly what events are emitted if you look in the ChupChop_Config.h, for example it wil show for the ChipChopEngine events: CC_SOCKET_CLOSED, CC_CONNECTED, CC_COMMAND_RECEIVED
Keep Alive has: KEEP_ALIVE_RESTARTING

And this is how you would use it:




//happens just before the KeepAlive will restart the esp
// you will add this callback to the even further down in the setup()
void keepAliveRestarting(){
    ///do something here, for example you can cancel the restart
    KeepAlive.cancelRestart();
    
    //do something clever

    KeepAlive.restart(); //continue the restart
    
}

// look in the setup() how this is called through events
void ChipChop_onCommandReceived(String target_component,String command_value, String command_source, int command_age){
Serial.println(target_component);
Serial.println(command_value);


}

void setup(){

    //option1. register a listening callback (the one above) with KeepAlive
    KeepAlive.addListener(KEEP_ALIVE_RESTARTING, keepAliveRestarting);

    //option2. or you can also write it as a lambda instead of the callback function keepAliveRestarting
    KeepAlive.addListener(KEEP_ALIVE_RESTARTING,[](){
        //do something

     });

    //ChipChop Engine example listeners, again you can write them as declared functions or as lambdas
    ChipChop.addListener(CC_CONNECTED,[](String a, String b, String c, int d){
        //the arguments a,b,c,d have no value but have to be declared, they are only used with CC_COMMAND_RECEIVED

    Serial.println(F("ChipChop socket is now open"));


    });

     ChipChop.addListener(CC_SOCKET_CLOSED,[](String a, String b, String c, int d){
Serial.println(F("ChipChop socket is closed"));
     });


    //this is an example of using events for the command received so instead of writing:
    // ChipChop.commandCallback(ChipChop_onCommandReceived);
    // you can write
    ChipChop.addListener(CC_COMMAND_RECEIVED, ChipChop_onCommandReceived);


}





If you want to try the Keep Alive you can either change the timeout value in the ChipChop_config.h to whatever you want

#define KEEP_ALIVE_TIMEOUT 10800000 - this would be 3 hours (I think :-)

and to have Keep Alive restart the esp

#define KEEP_ALIVE_MODE KEEP_ALIVE_AUTO_AGGRESSIVE
or
#define KEEP_ALIVE_MODE KEEP_ALIVE_RESTART

(don't ask me what the difference is, can't remember :-)

the option KEEP_ALIVE_AUTO will first try to restart the WiFi and then the esp so that's also a good one as if you have a weak WiFi signal restarting the WiFi modem can give it a small power boost to find a better signal

You will need to also include the ChipChopPlugins.h and the config, but it all takes very little space just generate a template code through the Code Builder and run it, I had to reprogram on of my esp8266 D1 minis and it's loaded with everything and that includes running the WiFi Portal which runs a webserver, dns server, html page and whatnot and still has 8kb to spare + it's scanning at 5ms intervals an I2C sensor

Have a look and shout if you need me :-)



TomaszB
22 Nov 2024

Hi Gizmo

Some results of my tests that included torturing scenario of WiFi connection / disconnection.

ESP initiated WiFi, then it was keeping connection 60 seconds, then it disconnected, waited 5 minutes and started cycle from the beginning.
I called ChipChop.pause() before disconnecting WiFi and ChipChop.unPause() on WiFi.onStationModeGotIP() that is called AFTER WiFi.onStationModeConnected() (so i hoped that this is final, ultimate event that confirm that connection is fully established and operational).

1) In such setup ESP restarted every time on the second WiFi.onStationModeGotIP() event. In this scenario i observed stack dump sent to a console. I added there a Ticker that delayed 5 seconds calling ChipChop.unPause() and the problem was gone.

2) I haven't noticed any apparent indicators of memory leaks. Value fluctuated a bit but after 17 hours it was on the similar level.

3) However, I noticed that ESP restatrs irregularly (1-2 times a day) but witohout dumping anything to console in debug mode (VisualStudio Code). It behaved silently, similarly to execution of ESP.restart() rather than when it runs out of memory where it dumps the stack to a console. I had connected console two times during such restart and it happaned after WiFi was disconnected (either instantly after turning WiFi off or after a 2-3 seconds). But the last observation can be a coincidence.

4) In this project stability is the priority, so now I'm checking how it behaves when WiFi is permanently connected.

But maybe you observed someting similar (presuming that you tested such dumb scenario....)


Gizmo
22 Nov 2024

hmmm, silent restarts after WiFi is off?

I haven't had such an issue exactly, if there is no memory dump then that sounds more like something to do with power. Bear in mind that there will be a power fluctuation when you switch the wifi modem/radio on and off, I know it spikes a fair bit when turning the wifi on but no idea what happens when you turn it off?

Also, I don't know how instantly the disposing of wifisecure.h actually happens, can you see if creating a small delay timer after you call pause() and before you turn the wifi off would help? I know you don't get a memory dump/error but I don't know if something may still try to use the wifi radio when it's actually powered down.

I don't have time right now to test but I will give it a go maybe over the weekend, I would probably also add in the setup() a Serial.print("restarted") and maybe also print the millis() before switching the wifi off just to see how long was actually the esp functioning before it rebooted as maybe there is some pattern.

oh, on a completely random note, I've just got a Raspberry PI Pico 2 and some w5500 ethernet module to work with ChipChop over LAN :-)



TomaszB
22 Nov 2024

Hi.

1) I'll try with this delay before switching off a Wifi but give me few days as now I'm running stability test in scenario where WiFi remains stable.

2) I think / hope that power stability is not an issue here. As you mentioned, it happens after disconnection, when you limit consumption. Moreover, it happened on a nodemcu board both with an external power supply and in case when it was connected to laptop USB port. So, restart could have happened once but not few times a day. Finally, it also happened a few seconds after switching wifi off.

The most important - have fun with Raspberry PI Pico 2 :-)

Gizmo
22 Nov 2024

I know you have to use 8266 but do you have an esp32 to run exactly same code and see if it behaves differently, just to make life easier to pinpoint if it's the hardware itself

i will try to run the test over the weekend and let you know next week