HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

parameterized Block

Placementjob -> parameterized

A parameterized job is used to encapsulate a set of work that can be carried out on various inputs much like a function definition. When the parameterized block is added to a job, the job acts as a function to the cluster as a whole.

The parameterized block allows job operators to configure a job that carries out a particular action, define its resource requirements and configure how inputs and configuration are retrieved by the tasks within the job.

To invoke a parameterized job, nomad job dispatch or the equivalent HTTP APIs are used. When dispatching against a parameterized job, an opaque payload and metadata may be injected into the job. These inputs to the parameterized job act like arguments to a function. The job consumes them to change its behavior, without exposing the implementation details to the caller.

To that end, tasks within the job can add a dispatch_payload block that defines where on the filesystem this payload gets written to. An example payload would be a task's JSON configuration.

Further, certain metadata may be marked as required when dispatching a job so it can be used to inject configuration directly into a task's arguments using interpolation. An example of this would be to require a run ID key that could be used to lookup the work the job is suppose to do from a management service or database.

Each time a job is dispatched, a unique job ID is generated. This allows a caller to track the status of the job, much like a future or promise in some programming languages. The dispatched job cannot be updated after dispatching; to update the job definition you need to update the parent job.

job "docs" {
  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
    meta_optional = ["pager_email"]
  }
}

See the multiregion documentation for additional considerations when dispatching parameterized jobs.

parameterized Requirements

parameterized Parameters

  • meta_optional (array<string>: nil) - Specifies the set of metadata keys that may be provided when dispatching against the job.

  • meta_required (array<string>: nil) - Specifies the set of metadata keys that must be provided when dispatching against the job.

  • payload (string: "optional") - Specifies the requirement of providing a payload when dispatching against the parameterized job. The maximum size of a payload is 16 KiB. The options for this field are:

    • "optional" - A payload is optional when dispatching against the job.

    • "required" - A payload must be provided when dispatching against the job.

    • "forbidden" - A payload is forbidden when dispatching against the job.

parameterized Examples

The following examples show non-runnable example parameterized jobs:

Required Inputs

This example shows a parameterized job that requires both a payload and metadata:

job "video-encode" {
  # ...

  type = "batch"

  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
  }

  group "encode" {
    # ...

    task "ffmpeg" {
      driver = "exec"

      config {
        command = "ffmpeg-wrapper"

        # When dispatched, the payload is written to a file that is then read by
        # the created task upon startup
        args = ["-config=${NOMAD_TASK_DIR}/config.json"]
      }

      dispatch_payload {
        file = "config.json"
      }
    }
  }
}

Metadata Interpolation

job "email-blast" {
  # ...

  type = "batch"

  parameterized {
    payload       = "forbidden"
    meta_required = ["CAMPAIGN_ID"]
  }

  group "emails" {
    # ...

    task "emailer" {
      driver = "exec"

      config {
        command = "emailer"

        # The campaign ID is interpolated and injected into the task's
        # arguments
        args = ["-campaign=${NOMAD_META_CAMPAIGN_ID}"]
      }
    }
  }
}

Interactions with periodic

When trying to schedule a job that is both parameterized and periodic, an internal hierarchy comes into play: A parametrized job should not be dispatched without parameters by the periodic configuration, causing the parameterized option to take precedence: Once the job is dispatched the first time, and given parameters, the periodic configuration comes into play and a new job job will be dispatched according to that job and not the originally submitted one.

For example a job with the following configuration: 

  periodic {
    crons = [
      "*/40 * * * * * *"
    ]
  }
  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
    meta_optional = ["pager_email"]
  }

Will not trigger any new jobs until it is dispatched at least once, and after that the dispatched child will periodically trigger more children with the given parameters:

$ nomad job status                                                      
ID                                                     Type                          Priority  Status   Submit Date
sync                                                   batch/periodic/parameterized  50        running  2024-11-07T10:43:30+01:00 // Original submitted job
sync/dispatch-1730972650-247c6e97                      batch/periodic                50        running  2024-11-07T10:44:10+01:00 // First dispatched job with parameters A
sync/dispatch-1730972650-247c6e97/periodic-1730972680  batch                         50        running  2024-11-07T10:44:40+01:00 // Cron job with parameters A
sync/dispatch-1730972650-247c6e97/periodic-1730972860  batch                         50        running  2024-11-07T10:47:40+01:00 // Cron job with parameters A
sync/dispatch-1730972760-f79a96e1                      batch/periodic                50        running  2024-11-07T10:46:00+01:00 // Second dispatched job with parameters B
sync/dispatch-1730972760-f79a96e1/periodic-1730972800  batch                         50        running  2024-11-07T10:46:40+01:00 // Cron job with parameters B
sync/dispatch-1730972760-f79a96e1/periodic-1730972860  batch                         50        running  2024-11-07T10:47:40+01:00 // Cron job with parameters B

In order to force a cron job, the corresponding parameterized job needs to be used:

$ nomad job periodic force sync/dispatch-1730972650-247c6e97
Edit this page on GitHub