Beanstalk

The Beanstalk source can be used to read content from Beanstalk queues. This, combined with the Beanstalk Operator, allows a full-circle workflow.

Configuration Options

• module (required): beanstalk
• paths (required): A list of XPath-like expressions representing the JSON fields you want to extract from.
• reference: An XPath-like expression representing the JSON field you want to use as a reference. (default: source name).
• host (required): Host to connect to.
• port (required): Port to connect over.
• queue_name (required): The name of the Beanstalk queue you want to use.

Example Configuration

Inside the sources section of your configuration file:

- name: beanstalk-input
  module: beanstalk
  paths: [content]
  reference: reference
  host: 127.0.0.1
  port: 11300
  queue_name: MYQUEUENAME

If you are expecting JSON jobs in the Beanstalk queue that look like this:

{
    "content": "freeform text",
    "reference": "http://example.com"
}

The above config will extract artifacts from the value of the content key, and use the value of the reference key as the artifact’s reference.

If you instead had JSON jobs like this:

{
    "data": {
        "text": "freeform text",
        "more": "more text",
        "ref": "http://example.com"
    }
}

And you want to extract from text and more, with ref as a reference, you could set up your config to account for the more complex JSON structure:

paths: [data.text, data.more]
reference: data.ref

This flexibility allows easier integration with arbitrary systems.