Hello,
I have been playing on ECC for a long while now, and I know the restarts are relatively new. But I have not heard of any reasons why there is all this lag (hence the need for restarts). I tried searching this forum but I couldn't find any actual technical reasons given for this issue. I am simply trying to understand why this issue has not been solved because it appears to me that the restarts treat the symptoms but not the underlying problem. I understand that ECC is volunteer run, and you guys have dedicated yourselves to keeping ECC up and running, working your butts off both in ECC and at your day jobs. Thanks for all you have done.

It seems to me with my limited knowledge of programming (I am an information security guy with a growing interest in Malware Analysis) that there is a process or program within the server that is filling a cache or possibly an entire memory space with useless data. So I have a couple of questions:

1) Have you been able to identify the offending process or service?
2) If not, are you able to obtain a memory dump of the server? FTK imager (free version) is well respected in the field of IT forensics, and will allow you to obtain an accurate memory dump that can then be examined for damaged or buggy process images.
3) Are there log files that pertain to this problem? (are you able to post a suitable redacted copy of said logs if applicable?)
4) Is this an issue with the underlying server OS (CentOS I am guessing?) or a minecraft plugin or server?

Like I said, I am learning assembly for my job and I do solid troubleshooting. If you think I can help just let me know!

Theory5

 

LOL.

But that's ridiculous. You can't just attribute this kind of problem to normal server activity. It seems like these restarts happen at LEAST once a day, which would be wholly unacceptable in any sort of business level network. Imagine if a company had to restart all its servers at least once a day. (I know, many admins do advocate restart schedules, but they are usually once a month at best (or once every couple of days at worst) and are always done during off-hours and cycle the systems, not all at once).

Also, logs are saved to disk, not memory. If logs are causing the lag, it could be indicative of too many verbose logs attempting to generate, which is pretty easily solved. One of the guys at my job set a network appliance (Orion) to do something like that, it ended up taking the appliance down.

Now, if you guys are managing to generate enough log files that it fills up the disk (which would mean you have a script that removals all logs after restart, rendering the amount of logs generated null, anyways) I would suggest an event management solution that is not on the server itself.

As for the memory itself, see my first post. OR if this is not the case, it could be indicative of a program failing to properly release its memory space once it is done, resulting in chunks of corrupted or orphaned (I think thats the right word) allocated spaces.

Have you guys done any sort of troubleshooting analysis yet?

 

Perhaps running MC through the EMET (if you guys have a WinTel server) might help. http://www.microsoft.com/en-us/download/details.aspx?id=29851.
This will force programs that do not take advantage of exploit mitigation services such as DEP to route through them. This may help in reducing undesired behavior (i.e. memory leak) as well. The only thing is this tool requires extensive testing before use on a production system.

Anyways, thanks for your prompt responses.
Theory5

 

Oops, I am sorry if I was not clear. A memory dump is a capture of everything that is happening in memory at that exact moment. When you run a program (say, MC) it does not run directly from your hard drive, instead it is put into memory (RAM) to run. From there the program might then write changes back to the hard drive (which is why you must click "save" in a word processor for it to actually save your work).

While a program is running (in memory) it may alter itself, so that it's "image" is different in memory than it is on the disk. Sometimes this is on purpose, but when there is a bug in a process or program it might change the image in memory in undesirable ways. The actual image on the disk is unaffected, but the runtime image (in memory) may be altered for better or for worse. I was saying that if a process has a bug (like a memory leak) you need to dump the memory (RAM) in order to see if an undesired operation altered the process in memory. You can also see what the program is interacting with. So if there are two plugins that do not work well together, their struggle might be seen in a memory dump.

Memory dumps are used for troubleshooting and forensic purposes.

 

Thanks Andrew, I will be sure to do that!