CPU cycles, or memory leak. Pick one!
Posted: 18 Feb 2014, 18:11As many people have noted, after running an EQ2 session for a long time (6-12 hours), it will eventually crash out, because it exceeds the 32-bit application memory limit. This is caused by a "memory leak". In short, a section of code uses memory, but doesn't delete it. These are bad, but only create really big issues, when that section of code is called over, and over, and over, and over...
Lets use an over simplified example:
A 32 bit application memory limit is 4gb (4,000,000 kb).
A section of code with a memory leak creates 100kb of memory, but doesn't delete it.
Lets say EQ2 uses 2 gb of memory on it's own for standard running.
Now, that bad section of code, is run once per SECOND. That means every single second, it's "wasting" 100kb of memory.
Our memory limit is 4,000,000, and EQ2 is already using 2,000,000. That means we have 2,000,000kb left.
If we're using 100kb every second, it means we will run out after approximately 20,000 seconds, or 5.5 hours.
Now, obviously the 'best' way to fix this, is to fix the bad section of code, so it doesn't leak memory. Unfortunately it's not always that easy. 1) They aren't always easy to find in code, and 2) It's not always in "your" code.
For example, EQ2 has a lot of code managed by different people how 'we' use it.
EQ2 - SOE
Innerspace - Lavish
ISXEQ2 - Amadeus
ISXOgre - Kannkor
ISXBJ - BJCasey
We managed to track down this memory leak to a specific, place in EQ2's code, and able to reproduce it (which verifies what program it's actually in).
So, back on track. A long time ago, SOE started adding awesomium_process.exe to run when EQ2 was run. This did something with the in-game browser. The problem was, they were loading MULTIPLE of these for every single EQ2 session. Some times people would have 12 of these running. For most people, we didn't use the in-game browser, so I made ISXOgre just delete the file, so it wouldn't run and eat up CPU cycles.
Unfortunately, when this process is removed, EQ2 continually tries to re-create it. It's this re-creating it that has a memory leak. So every second, EQ2 is trying to run awesomium_process.exe, and it's leaking memory. Given enough time, the session crashes.
So that's the choice. Have awesomium_process.exe eat CPU cycles, or have a memory leak. However, it may not be quite that simple, because if it's trying to create awesomium_process.exe, then it's using CPU cycles to do that also.
Anyways, long story short, I'm going to make ISXOgre not automatically delete awesomium_process.exe going forward, to prevent the EQ2 memory leak. I'm going to add a new option to the ISXOgre config (type into the console: ogre config) that will give you the option to have it deleted; with a warning, that a memory leak will start.
For those curious, here are a few tests you can run to see results:
On the 'ogre config' screen, make sure (at the time of writing this, before I change the options) is CHECKED: "Do not remove awesomium_process".
Patch EQ2 OR copy awesomium_process.exe back into your Everquest2/ directory.
You can stay at the loginscene for all these tests.
Load 1 EQ2 with NO autoloads. Once it's fully loaded, monitor memory usage (task manager). It should be relatively stable.
ext isxeq2
Memory will increase to load ISXEQ2, then become stable.
ext isxogre
Memory will increase to load ISXOgre, then become stable.
No memory leak present.
Now, close your Everquest2 session down.
Rename/delete awesomium_process.exe
Load 1 EQ2 with NO autoloads. Memory is never stable, and has a memory leak.
At this stage, it doesn't really matter if you load isxeq2/isxogre, the memory leak already exists, so it will continue to exist.
Special thanks to macker for finding this process had this effect.
Lets use an over simplified example:
A 32 bit application memory limit is 4gb (4,000,000 kb).
A section of code with a memory leak creates 100kb of memory, but doesn't delete it.
Lets say EQ2 uses 2 gb of memory on it's own for standard running.
Now, that bad section of code, is run once per SECOND. That means every single second, it's "wasting" 100kb of memory.
Our memory limit is 4,000,000, and EQ2 is already using 2,000,000. That means we have 2,000,000kb left.
If we're using 100kb every second, it means we will run out after approximately 20,000 seconds, or 5.5 hours.
Now, obviously the 'best' way to fix this, is to fix the bad section of code, so it doesn't leak memory. Unfortunately it's not always that easy. 1) They aren't always easy to find in code, and 2) It's not always in "your" code.
For example, EQ2 has a lot of code managed by different people how 'we' use it.
EQ2 - SOE
Innerspace - Lavish
ISXEQ2 - Amadeus
ISXOgre - Kannkor
ISXBJ - BJCasey
We managed to track down this memory leak to a specific, place in EQ2's code, and able to reproduce it (which verifies what program it's actually in).
So, back on track. A long time ago, SOE started adding awesomium_process.exe to run when EQ2 was run. This did something with the in-game browser. The problem was, they were loading MULTIPLE of these for every single EQ2 session. Some times people would have 12 of these running. For most people, we didn't use the in-game browser, so I made ISXOgre just delete the file, so it wouldn't run and eat up CPU cycles.
Unfortunately, when this process is removed, EQ2 continually tries to re-create it. It's this re-creating it that has a memory leak. So every second, EQ2 is trying to run awesomium_process.exe, and it's leaking memory. Given enough time, the session crashes.
So that's the choice. Have awesomium_process.exe eat CPU cycles, or have a memory leak. However, it may not be quite that simple, because if it's trying to create awesomium_process.exe, then it's using CPU cycles to do that also.
Anyways, long story short, I'm going to make ISXOgre not automatically delete awesomium_process.exe going forward, to prevent the EQ2 memory leak. I'm going to add a new option to the ISXOgre config (type into the console: ogre config) that will give you the option to have it deleted; with a warning, that a memory leak will start.
For those curious, here are a few tests you can run to see results:
On the 'ogre config' screen, make sure (at the time of writing this, before I change the options) is CHECKED: "Do not remove awesomium_process".
Patch EQ2 OR copy awesomium_process.exe back into your Everquest2/ directory.
You can stay at the loginscene for all these tests.
Load 1 EQ2 with NO autoloads. Once it's fully loaded, monitor memory usage (task manager). It should be relatively stable.
ext isxeq2
Memory will increase to load ISXEQ2, then become stable.
ext isxogre
Memory will increase to load ISXOgre, then become stable.
No memory leak present.
Now, close your Everquest2 session down.
Rename/delete awesomium_process.exe
Load 1 EQ2 with NO autoloads. Memory is never stable, and has a memory leak.
At this stage, it doesn't really matter if you load isxeq2/isxogre, the memory leak already exists, so it will continue to exist.
Special thanks to macker for finding this process had this effect.