Pause Laravel queue execution

I’ve been building a new side project that queries the GitHub API a lot; enough that I’m likely to hit my rate limit semi-regularly.

Each task that needs the GitHub API is dispatched as a job, which means that so long as I can detect when my rate limit has been exceeded I can stop a specific queue from processing new jobs until my rate limit is reset.

Stop the queue running

Before Laravel fetches a job for a queue it runs the Queue::looping callback to see if the job can proceed. You can register a new callback for Queue::looping in the boot method of app/Providers/EventServiceProvider.php:

php
<?php
namespace App\Providers;
use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Queue;
class EventServiceProvider extends ServiceProvider
{
    public function boot()
    {
        Queue::looping(function (\Illuminate\Queue\Events\Looping $event) {
            // If there's a circuit breaker set on the github queue, don't execute the job
            if (($event->queue == 'github') && (Cache::has('github-rate-limit-exceeded'))) {
                return false;
            }
            return true;
        });
    }
}

If the method returns true the job will run as expected. If it returns false, no job will be processed and the worker will wait until the next loop (approx 3 seconds).

In this example I check a specific queue name (github), which means the circuit breaker only affects jobs in that queue and not any others. This means that I need to run MyJob::dispatch($params)->onQueue('github'); to dispatch a job and php artisan queue:work --queue github to process the events.

Handle rate limit errors

Now that there’s a circuit breaker in place it’s time to add the code to my job that sets the github-rate-limit-exceeded cache key if required.

The Laravel-GitHub package throws a RuntimeException when a rate limit error occurs, so we need to catch that and use the rate_limit API to check if we’re out of API requests. If we are, we use Cache::add to set the key used by the circuit breaker.

Putting all that together, you get the following handle method from my \App\Jobs\QueryGitHub job:

php
public function handle()
{
    try {
        // You'll probably be doing something more interesting, but this
        // uses up my rate limit enough to test
        dump(GitHub::me()->organizations());
    } catch (\Github\Exception\RuntimeException $e) {
        // If there's an exception, check our rate limit
        $limits = GitHub::api('rate_limit')->getResources();
        $reset = $limits['core']->getReset();
        // If there are no more requests available, add a cache entry
        if ($limits['core']->getRemaining() <= 0) {
            Cache::add('github-rate-limit-exceeded', $reset, ($reset - time()));
        }
        // Rethrow the exception to mark the job as failed
        throw $e;
    }
}

That’s all!

Just 25 lines of code and I have a rate-limit aware job queue that will stop executing until my GitHub rate limit resets. If you’d like to see the entire set of changes in a single diff it’s available on GitHub