Fatal error using Requests::request_multiple

  • sil3r

    (@sil3r)


    7 years, 7 months ago

    Hello,

    first, thank you for taking the time to read me.

    I wrote a plugin, doing in average one thousand requests each 5 minutes, using cron. It’s a lot I know but it’s not to brute force a form, DOS a server or spam. It’s only to crawl the Apple App Store to detect prices drops, updates and new releases using their API.

    I disabled cron to execute it server side to be sure to not impact visitors. Using define('DISABLE_WP_CRON', true);

    Everything works perfectly BUT sometimes my cron script return an error, hopefully cron notified me by email :

    PHP Warning:  require_once(/home/www/wp-includes/Requests/Response.php): failed to open stream: Too many open files in /home/www/wp-includes/class-requests.php on line 146
PHP Fatal error:  require_once(): Failed opening required '/home/www/wp-includes/Requests/Response.php' (include_path='.:/usr/user/php/7.2.9/lib/php') in /home/www/wp-includes/class-requests.php on line 146

Warning: require_once(/home/www/wp-includes/Requests/Response.php): failed to open stream: Too many open files in /home/www/wp-includes/class-requests.php on line 146

Fatal error: require_once(): Failed opening required '/home/www/wp-includes/Requests/Response.php' (include_path='.:/usr/user/php/7.2.9/lib/php') in /home/www/wp-includes/class-requests.php on line 146

    This error does not occurs every time my cron script is executed, but only some times (~1 time each 2 hours). If someone can help me to debbug this problem, that would be awesome.
    FYI, I already read some things about ulimit, but I don’t want to change it.

    Thanks for your time, sorry for my english I’m french.
    Sil3r

Viewing 3 replies - 1 through 3 (of 3 total)
  • Moderator bcworkz

    (@bcworkz)

    7 years, 7 months ago

    There are a number of possible open file limits besides the per user limit (ulimit). Any one of them could be the cause. Assuming they all are set to reasonable levels, the file limit issue is often due to poor coding practice where opened files are not promptly closed when the file is done with. This issue can be less than obvious when looping and recursive code is involved. Be sure none of your code is abusing the file system in this way. If that checks out, verify the file system limits are reasonable. ulimit is only one aspect of this.

    Thread Starter sil3r

    (@sil3r)

    7 years, 7 months ago

    Hello,

    thanks for your reply but I never open/write any file in my script/loop, I only use Requests::request_multiple to fetch JSON data and compare it to my database using wpdb->query.
    I suspect Requests::request_multiple or Curl to put the header or the body into a file in /tmp folder.

    FYI, this error does not occurs if I use php function file_get_contents, but by using this function my script is no longer async and much slower to be executed (10 secondes VS 4 minutes).

    Regards

    • This reply was modified 7 years, 7 months ago by sil3r.
    Moderator bcworkz

    (@bcworkz)

    7 years, 7 months ago

    Hmmm. I don’t think Requests normally writes to files unless it is explicitly told to do so in the options passed. Maybe if PHP is running short of memory, some of it is paged out to the file system? Could you perhaps reduce the number of requests made through request_multiple()?

    It seems server resources are being maxed out for whatever reason. Your only options are either to reduce the load or increase resources. Of course how that is done covers a broad range from a mere setting to better hardware. Without knowing any specifics, it appears to me like reducing requests is what you have the most control over.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Fatal error using Requests::request_multiple’ is closed to new replies.