Neos CMS Meilisearch indexer based on a job queue

This package can be used to index a huge amount of nodes in Meilisearch indexes. This package use the Flowpack JobQueue packages to handle the indexing asynchronously.

Topics

• Installation
• Indexing
• SupervisorD configuration
• Update Instructions

Installation and Configuration

You need to install the correct Queue package based on your needs.

Available packages:

• sqlite
• beanstalkd
• doctrine
• redis

Please check the package documentation for specific configurations.

The default configuration uses the FakeQueue, which is provided by the JobQueue.Common package. Note that with that package jobs are executed synchronous with the flow nodeindexqueue:build command.

Check the Settings.yaml to adapt based on the Queue package, you need to adapt the className:

Flowpack:
  JobQueue:
    Common:
      presets:
        'Medienreaktor.Meilisearch.ContentRepositoryQueueIndexer':
          className: 'Flowpack\JobQueue\Common\Queue\FakeQueue'

If you use the doctrine package you have to set the tableName manually:

Flowpack:
  JobQueue:
    Common:
      presets:
        'Medienreaktor.Meilisearch.ContentRepositoryQueueIndexer':
          className: 'Flowpack\JobQueue\Doctrine\Queue\DoctrineQueue'
      queues:
        'Medienreaktor.Meilisearch.ContentRepositoryQueueIndexer':
          options:
            tableName: 'flowpack_jobqueue_QueueIndexer'
        'Medienreaktor.Meilisearch.ContentRepositoryQueueIndexer.Live':
          options:
            tableName: 'flowpack_jobqueue_QueueIndexerLive'

Indexing

Batch Indexing

How to build indexing jobs

flow nodeindexqueue:build --workspace live

Optional parameters:

--start-node-path="/sites/yoursitename" Limit job creation to the given node path (the subtree starting at that node will be indexed).

--dimensions-hash="e781f29c8dd927c09735547a848e3459" Restrict indexing to a single DimensionSpacePoint (e.g. a specific language / market combination). The hash must match an existing configured dimension combination.

Combined example:

flow nodeindexqueue:build --workspace live --start-node-path="/sites/yoursitename" --dimensions-hash="e781f29c8dd927c09735547a848e3459"

You can use this CLI command to process indexing job:

flow nodeindexqueue:work --queue batch

Live Indexing

You can disable async live indexing by editing Settings.yaml:

Medienreaktor:
  Meilisearch:
    ContentRepositoryQueueIndexer:
      enableLiveAsyncIndexing: false

You can use this CLI command to process indexing job:

flow nodeindexqueue:work --queue live

Live Indexing via DB

You have to run doctringe:migrate in order to create a lastChecked-Entry in the DB. Then you can enable async indexing via db by editing Settings.yaml:

Medienreaktor:
  Meilisearch:
    ContentRepositoryQueueIndexer:
      enableAsyncDBIndexing: true
      enableLiveAsyncIndexing: false

You can use this CLI command to generate indexing job:

flow nodeindexqueue:indexchangednodes --workspace=live --exit-after=120

Create Jobs for changed nodes

You should use a cronjob like this, for working:

*/5 * * * * FLOW_CONTEXT=Production /var/www/flow nodeindexqueue:indexchangednodes --workspace=live --exit-after=120 */5 * * * * FLOW_CONTEXT=Production /var/www/flow nodeindexqueue:work --queue live --exit-after=120 --verbose=0

License

Licensed under MIT, see LICENSE