Pause Laravel queue execution

12 Apr 2021 in Development

Use a circuit breaker to stop Laravel job queue execution when you hit your GitHub rate limit

I’ve been building a new side project that queries the GitHub API a lot; enough that I’m likely to hit my rate limit semi-regularly.

Each task that needs the GitHub API is dispatched as a job, which means that so long as I can detect when my rate limit has been exceeded I can stop a specific queue from processing new jobs until my rate limit is reset.

Stop the queue running

Before Laravel fetches a job for a queue it runs the Queue::looping callback to see if the job can proceed. You can register a new callback for Queue::looping in the boot method of app/Providers/EventServiceProvider.php:

<?php

namespace App\Providers;

use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Queue;

class EventServiceProvider extends ServiceProvider
{
public function boot()
{
Queue::looping(function (\Illuminate\Queue\Events\Looping $event) {
// If there's a circuit breaker set on the github queue, don't execute the job
if (($event->queue == 'github') && (Cache::has('github-rate-limit-exceeded'))) {
return false;
}

return true;
});
}
}

If the method returns true the job will run as expected. If it returns false, no job will be processed and the worker will wait until the next loop (approx 3 seconds).

In this example I check a specific queue name (github), which means the circuit breaker only affects jobs in that queue and not any others. This means that I need to run MyJob::dispatch($params)->onQueue('github'); to dispatch a job and php artisan queue:work --queue github to process the events.

Handle rate limit errors

Now that there’s a circuit breaker in place it’s time to add the code to my job that sets the github-rate-limit-exceeded cache key if required.

The Laravel-GitHub package throws a RuntimeException when a rate limit error occurs, so we need to catch that and use the rate_limit API to check if we’re out of API requests. If we are, we use Cache::add to set the key used by the circuit breaker.

Putting all that together, you get the following handle method from my \App\Jobs\QueryGitHub job:

public function handle()
{
try {
// You'll probably be doing something more interesting, but this
// uses up my rate limit enough to test
dump(GitHub::me()->organizations());
} catch (\Github\Exception\RuntimeException $e) {
// If there's an exception, check our rate limit
$limits = GitHub::api('rate_limit')->getResources();
$reset = $limits['core']->getReset();

// If there are no more requests available, add a cache entry
if ($limits['core']->getRemaining() <= 0) {
Cache::add('github-rate-limit-exceeded', $reset, ($reset - time()));
}

// Rethrow the exception to mark the job as failed
throw $e;
}
}

That’s all!

Just 25 lines of code and I have a rate-limit aware job queue that will stop executing until my GitHub rate limit resets. If you’d like to see the entire set of changes in a single diff it’s available on GitHub