Developed with love by KnpLabs Hire us for your project!
29

HeriJobQueueBundle

by heristop

This Symfony bundle provides the use of Zend Queue from Zend Framework

JobQueueBundle

SensioLabsInsight Scrutinizer Code Quality Build Status

This bundle provides the use of Zend Queue from Zend Framework. It allows your Symfony 2/3 application to schedule multiple console commands as server-side jobs.

See the Programmer's Reference Guide for more information.

Features:

  • Manage multiple queues
  • Schedule your Symfony commands
  • Monitor messages and log exceptions
  • Retry logic
  • Prioritizing jobs
  • RabbitMQ or database support
  • Easy-to-use :)

Installation

Require heristop/jobqueue-bundle to your composer.json file:

{
    "require": {
        "heristop/jobqueue-bundle": "1.0.1",
        "zendframework/zend-queue": "2.*@dev"
    }
}

Load the bundle in AppKernel:

    $bundles[] = new Heri\Bundle\JobQueueBundle\HeriJobQueueBundle();

If you use the Doctrine adapter, update your database:

    app/console doctrine:schema:update --force

If you use the Amqp adapter, you may configure the connection in this way:

    # app/config/config.yml
    heri_job_queue:
        amqp_connection:
            host: localhost
            port: 5672
            user: guest
            password: guest

Usage

First, define a message which contains the Symfony command to call. For instance, we choose to add the clear command in a queue named "queue1":

    $queue = $this->get('jobqueue');
    $queue->attach('queue1');

    $queue->push([
        'command' => 'cache:clear'
    ]);

You can also call commands with arguments:

    $queue->push([
        'command'   => 'demo:great',
        'argument'  => [
            'name'   => 'Alexandre',
            '--yell' => true
        ]
    ]);

Then, add the queue to listen in the configuration:

    # app/config/config.yml
    heri_job_queue:
        enabled:            true
        max_messages:       1
        # set a process_timeout (in seconds) if you want your commands to be killed
        # when their execution time is too long (default: null)
        process_timeout:    60
        queues:             [ queue1 ]

Note: The queue is automatically created, but you can also use the command-line interface in this way:

    app/console jobqueue:create queue1

Listener Commands

Run the Listener

To run new messages pushed into the queue, execute this command:

    app/console jobqueue:listen

Specify a specific Queue

You may specify which queue connection the listener should utilize (skipping configuration):

    app/console jobqueue:listen queue1

Specify the Sleep Duration

You may also specify the number of seconds to wait before polling for new jobs:

    app/console jobqueue:listen --sleep=5

Process the first Job on the Queue

To process only the first job on the queue, you may use the jobqueue:work command:

    app/console jobqueue:work

Show Jobs

To see the pending jobs, run the command below:

    app/console jobqueue:show [queue-name]

Failed Jobs

If a job failed, the exception is logged in the database, and the command is call again after the setted timeout (default 90 seconds):

ScreenShot

To delete all of your failed jobs, you may use the jobqueue:flush command:

    app/console jobqueue:flush [queue-name]

Jobs Priority

Jobs are executed in the order in which they are scheduled (assuming they are in the same queue). You may also prioritize a call:

    $queue
        ->highPriority()
        ->push([
            'command' => 'cache:clear',
        ]);

Jobs Monitoring

If you use the Doctrine Adapter, you may use Sonata Admin to monitor your jobs:

    #
    # more information can be found here http://sonata-project.org/bundles/admin
    #
    sonata_admin:
        # ...
        dashboard:
            # ...
            groups:
                # ...
                System:
                    label:           System
                    icon:            '<i class="fa fa-wrench"></i>'
                    items:
                        - admin.queue

ScreenShot

Retry strategy

By default, number of excecution of failed messages is endless. If you use the Doctrine Adapter you may edit the max number of retries on queue table.

To retry all of your failed jobs, you may use this command:

    app/console jobqueue:retry [queue-name]

If you would like to delete a failed job, you may use this command:

    app/console jobqueue:forget [id]

Configure a daemon

The jobqueue:listen command should be runned with the prod environnement and the quiet option to hide output messages:

    app/console jobqueue:listen --env=prod --quiet

To avoid a memory leak caused by the monolog fingers crossed handler, you may configure the limit buffer size on config_prod.yml:

    # app/config/config_prod.yml
    monolog:
        handlers:
            main:
                type:         fingers_crossed
                action_level: error
                handler:      nested
                buffer_size:  50

Linux ProTip:

To run the command as a service, edit jobqueue-service shell in Resources/bin.
Set the correct PROJECT_ROOT_DIR value, and copy this file to /etc/init.d.

Then use update-rc.d:

    cp jobqueue-service /etc/init.d/jobqueue-service
    cd /etc/init.d && chmod 0755 jobqueue-service
    update-rc.d jobqueue-service defaults

To remove the service, use this command:

    update-rc.d -f jobqueue-service remove

If the service stopped suddenly, you may use supervisord to restart it automatically.

A sample config might look like this:

  [program:jobqueue-service]
  command=/usr/bin/php %kernel.root_dir%/console jobqueue:listen --env=prod
  directory=/tmp
  autostart=true
  autorestart=true
  startretries=3
  stderr_logfile=/var/log/jobqueue-service/jobqueue.err.log
  stdout_logfile=/var/log/jobqueue-service/jobqueue.out.log
  user=www-data
~
The MIT License (MIT)

Copyright (c) 2013-2016 Alexandre Mogère

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
heri_job_queue:
adapter: doctrine
enabled: true
max_messages: 1
process_timeout: ~
queues: []
amqp_connection:
host: localhost
port: 5672
user: guest
password: guest
  • Updated README
    By web-flow, 2 years ago
  • [Insight] Project files should not mix end of lines
    By heristop, 2 years ago
  • [PHP-CS] Applied fix
    By heristop, 2 years ago
  • Added LICENSE
    By heristop, 2 years ago
  • Merge pull request #24 from Madlexx/master
    By web-flow, 2 years ago
  • Merge pull request #1 from Madlexx/Madlexx-queue_service-path
    By web-flow, 2 years ago
  • Update QueueService.php
    By web-flow, 2 years ago
  • [Doctrine] Allow disabling retry logic if maxRetry is setted to 0
    By heristop, 2 years ago
  • [Travis] Removed failure section
    By Alexandre Mogère, 2 years ago
  • [Travis] Updated config
    By heristop, 2 years ago
  • [Travis] Updated allowed failures
    By heristop, 2 years ago
  • [Travis] Added SF3.1 && SF3.2 in matrix
    By heristop, 2 years ago
  • [Admin] Fixed num retries display
    By heristop, 2 years ago
  • [Doctrine] Added retry and forget commands
    By heristop, 2 years ago
  • [Travis] Use php-amqplib to test RabbMQ
    By heristop, 2 years ago
  • [Composer] Fixed zq version
    By heristop, 2 years ago
  • PHP-CS-FIXER
    By heristop, 2 years ago
  • [Admin][Message] Added num retries fields
    By heristop, 2 years ago
  • [Composer] Replaced videlalvaro/php-amqplib suggestion, fixes #23
    By heristop, 2 years ago
  • [CodeReview] PHP-CS-FIXER
    By Alexandre Mogère, 2 years ago
  • Fixed requirement
    By Alexandre Mogère, 3 years ago
  • Merge branch 'master' of https://github.com/heristop/HeriJobQueueBundle
    By heristop, 3 years ago
  • Removed scrutinizer config
    By heristop, 3 years ago
  • Merge pull request #22 from jlirochon/fix-process-timeout
    By heristop, 3 years ago
  • fixes processTimeout not updated when calling setUp()
    By , 3 years ago
  • Updated travis config
    By heristop, 3 years ago
  • Removed SF 2.3 support, Added SF 3.0 support
    By heristop, 3 years ago
  • Fixed travis configuration
    By heristop, 3 years ago
  • Updated composer.json
    By heristop, 3 years ago
  • Fixed typo
    By heristop, 3 years ago