How do I manipulate data to pass to the next task?

Thanks for y’alls patience as I teach myself YAQL and Orquesta…

I now have a semi-fuinctioning workflow. Here’s where I’m stuck:

I’m publishing dictionaries of result() data from tasks into variables. Now I want to use part of that information as input to a pre-defined task in a pack (AWS in this case).

The task has two input variables. The data I need for one of those input variables has to be derived by identifying one value from a dictionary, and then looking up another dictionary value in a list that has an element matching that value, and then returning the desired ID string. Logic dictates I would declare a variable like:

  dxcon_parse:
    action: core.noop
    input:
      dxlist: <% dxcon_info().get("connections") %>
      dxcons: <% dict(dxlist=>dict(ctx(dxlist).select([$.vlan, $]))) %>
      aws_vlan: <% ctx(pf_vc_info).get(pf_csp_vlan) %>
      aws_con_id: <% dxcons().get(aws_dx_con_vlan) %>
    next:
      - do:
          - dxcon_accept
      - publish:
        - aws_dx_con_vlan: <% ctx(aws_vlan) %>
        - aws_dx_con_id: <% ctx(aws_con_id) %>

Syntactically I believe that is proper. However, my problem is that when I execute, the engine complains:

  "errors": [
    {
      "type": "content",
      "message": "Action \"aws.directconnect_confirm_connection\" has unexpected input \"aws_vlan\".",
      "spec_path": "tasks.dxcon_accept.input.aws_vlan",
      "schema_path": "properties.tasks.patternProperties.^\\w+$.properties.input.patternProperties.^\\w+$"
    }

It seems I can’t add new input variables, and I can’t (or don’t know how) declare derived variables that eventually feed the input of the task. I tried making up a new task, but it complained there was no action, and I tried using core.noop but it similarly complained of unexpected inputs.

Surely I can’t be the only person ever to need to massage some data out of one variable to feed into another?

Look closely at the error message. It’s not complaining that it cannot dig out and render the values from the context variables you’ve published, it’s complaining that there is no aws_vlan parameter to the aws.directconnect_confirm_connection action specified in/by the dxcon_accept task.

It’s the equivalent of this Python code:

>>> def dxcon_accept(arg1, arg2):
...     pass
...
>>> dxcon_accept(aws_vlan=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: dxcon_accept() got an unexpected keyword argument 'aws_vlan'

I took a look at that action in the AWS pack, and I didn’t see an aws_vlan parameter to that action, or the python-script action runner:

---
description: 
pack: aws
enabled: true
entry_point: run.py
name: directconnect_confirm_connection
runner_type: python-script
parameters:
  account_id:
    type: string
  region:
    type: string
  action:
    default: confirm_connection
    immutable: true
    type: string
  cls:
    default: directconnect
    type: string
  headers:
    type: string
  module_path:
    default: boto3
    immutable: true
    type: string
  connectionId:
    type: string
    description: ""
    required: true

Can you post the dxcon_accept task as well? That might help us figure out what parameter you should be using.

Also, the AWS pack sees a lot of updates. What version of the AWS pack are you running?

I’m using v2.0.0 of the aws pack.

Here’s the full workflow with some bits redacted:

version: 1.0

input:
  - aws_account_id: '111111111111'
  - fabric_onprem_vlan: '1111'

vars:
  - dxlist: null
  - dxcons: null
  - aws_vlan: null
  - aws_con_id: null

tasks:
  # [226, 53]
  fabric_query:
    action: fabric.get_info
    next:
      # #d14c83
      - do:
          - dxcon_create
        when: succeeded()
        publish:
          - fabric_info: <% result().result %>
  # [76, 203]
  dxcon_create:
    action: fabric.create_aws_conn
    input:
      log_level: DEBUG
      account_uuid: <% ctx(fabric_info).get(account_uuid) %>
      pop: CHI11
      aws_account_id: <% ctx(aws_account_id) %>
      connection_desc: pfdemo-cst_de1_to_aws
      fabric_port: <% ctx(fabric_info).get(port_circuit_id) %>
      speed: 50Mbps
      vlan: <% ctx(fabric_onprem_vlan) %>
      zone: A
    next:
      # #fd9d32
      - do:
          - dxcon_query
        when: succeeded()
        publish:
          - fabric_vc_info: <% result().result %>
  # [226, 353]
  dxcon_query:
    action: aws.directconnect_describe_connections
    next:
      - do:
          - dxcon_parse
        publish:
          - dxcon_info: <% result().result[0] %>
  # [26, 553]
  dxcon_parse:
    action: core.noop
    input:
      dxlist: <% dxcon_info().get("connections") %>
      dxcons: <% dict(dxlist=>dict(ctx(dxlist).select([$.vlan, $]))) %>
      aws_vlan: <% ctx(fabric_vc_info).get(fabric_csp_vlan) %>
      aws_con_id: <% dxcons().get(aws_dx_con_vlan) %>
    next:
      - do:
          - dxcon_accept
      - publish:
        - aws_dx_con_vlan: <% ctx(aws_vlan) %>
        - aws_dx_con_id: <% ctx(aws_con_id) %>
  # [126, 753]
  dxcon_accept:
    action: aws.directconnect_confirm_connection
    input:
      log_level: DEBUG
      account_id: <% ctx(aws_account_id) %>
      connectionId: <% ctx(aws_dx_con_id) %>

output:
  - success: "yay, it worked"

So I’m publishing the values I want to use as inputs to the aws.directconnect_confirm_connection action in the dxcon_parse task. The problem is that it won’t actually execute anything because I’m making up variables in the parse task that core.noop isn’t expecting and so it pukes. I don’t know how to manipulate values in serial and then pass them to the next task…thats the crux of what I’m trying to accomplish.

So I worked out how to properly pass the data… I need to publish the result or derived values as exit criteria from whatever process is acquiring the source data. Got it. Cool.

My new problem is that the raw YAQL interpreter properly produces the string value I need derived from the list of dictionary objects. Unfortunately the Orquesta engine does NOT produce the same result. In other words the YAQL syntax is valid, but isn’t working in the engine. This smells like an expression evaluation bug with Orquestra that I will open an Issue for.