MySQL spawns rogue processes

  • EcoCatLady

    (@ecocatlady)


    9 years, 6 months ago

    I have an intermittent problem on a wordpress site where periodically something triggers hundreds of MySQL processes as well as corresponding HTTP processes. The symptoms are that the site becomes inaccessible on both the front and back ends. Sometimes it just times out, other times it throws an “Error Establishing Database Connection.” The problem is sometimes triggered by uploading a photo, but other times it just seems to happen randomly. I have not found a way to trigger the problem with any consistency, so it’s very difficult to troubleshoot. Sometimes the site will crash multiple times per day, other times it will work fine for several weeks. It’s not a high traffic site, probably averages around 2-3K page impressions per day.

    The site is hosted on a dedicated server running Apache 2.2.15, MySQL 5.0.67 and Php 5.2.13. There are no other installations of WordPress or any other programs using MySQL. There is another site hosted on the server which does not use either WordPress or MySQL, and when the problem occurs the non-wordpress site works fine, though very slow.

    When the problem happens, if I look at the Service Monitor on my server I can see that there are literally hundreds of both MySQL and Apache processes running. If I restart both MySQL and Apache the problem goes away. It’s not a build up of processes either – it’s sudden – one minute there will be half a dozen processes, the next there are hundreds. It’s like something just goes crazy and starts spawning processes. When I look at the Apache logs there’s nothing out of the ordinary other than requests that timed out or were very slow to process.

    I’ve upgraded to the latest version of WordPress, upgraded all themes and plugins, switched themes, replaced all plugins with different ones to rule out a plugin problem (only one I haven’t replaced is the Configurable Tag Cloud), deleted all unused themes and plugins, run a malware scan, and optimized and repaired all tables in the database. The site has comments turned off and always has.

    I am completely flummoxed, and just not sure which direction to look. Could it be a hacker? A problem with MySQL? Problem with the database? Problem with WordPress? Denial of Service attack?

    Any suggestions would be greatly appreciated.

Viewing 7 replies - 1 through 7 (of 7 total)
  • Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    9 years, 6 months ago

    Look at your httpd access logs at the times this starts. Anything interesting going on then?

    Thread Starter EcoCatLady

    (@ecocatlady)

    9 years, 6 months ago

    Nothing out of the ordinary in the apache access logs – just typical GET requests, but the apache error logs show hundreds of lines that read:

    [Sat Nov 12 15:04:12 2016] [warn] child process 19108 still did not exit, sending a SIGTERM

    the process number increments 19490, then it throws this line for each process sequentially:

    [Sat Nov 12 15:04:13 2016] [error] child process 19108 still did not exit, sending a SIGKILL

    But Googling it looks like that could be from restarting Apache… so probably not the cause of the problem.

    Thread Starter EcoCatLady

    (@ecocatlady)

    9 years, 6 months ago

    p.s. I think mysql logs might be more helpful… except that I haven’t figured out how to enable them, and my research suggests this can be dangerous as it can really slow things down. Do you have any experience with mysql logs?

    Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    9 years, 6 months ago

    Anything interesting in /var/log/messages?

    Thread Starter EcoCatLady

    (@ecocatlady)

    9 years, 6 months ago

    Well… I don’t really know what’s supposed to be there. The log shows a time adjustment every hour on the hour, and an entry for every time an FTP connection was established – some of the FTP lines show my home IP address (I FTPd some theme changes this morning), others show unfamiliar IPs not sure what that means… I did upagrade a few plugins… but none of the “not me” FTP requests are around the time of the crash.

    The only thing around the time of the crash was this line:

    Nov 12 14:44:01 server named[3128]: client 183.56.172.145#20000: query (cache) ‘760291168.www.baidu.com/A/IN’ denied

    I have no idea what it means.

    Moderator Steven Stern (sterndata)

    (@sterndata)

    Volunteer Forum Moderator

    9 years, 6 months ago

    interesting.. baidu is the Chinese search engine. It may have done a big crawl of your site, overloading it.

    ONe thing: I notice you said you’re using PHP 5.2. Really, or is that a typo? If really, upgrade your PHP to at least 5.6 ASAP. MySQL should be 5.5 or better.

    Thread Starter EcoCatLady

    (@ecocatlady)

    9 years, 6 months ago

    That’s a great suggestion. I’ve been leery of upgrading because last time I did it caused major problems on the server – I ended up with incompatible versions of php and apache and all hell broke loose. I’m actually thinking of migrating to a new server altogether (been on this one for years now) so that is probably my next step.

Viewing 7 replies - 1 through 7 (of 7 total)

The topic ‘MySQL spawns rogue processes’ is closed to new replies.