Orquesta Workflow Passing JSON values

Hello all. As I am new to Stackstorm (v.3.2) and working to create workflows in orquesta, I’m struggling figure out how to debug and get values dynamically passed from one task to another. I’ve read through the documentation and looked through many of the examples; however, I’m not finding the right information to help me pass a particular JSON value that is received from the second task “GetTenantGroup”, into a variable (tgroupid), that is to be used in a third task “CreateTenant”

input:

• customername
• customerslug

vars:

• tgroupid: 0

tasks:
CreateTenantGroup:
action: netbox.post.tenancy.tenant_groups name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>
next:
- when: <% succeeded() %>
do: GetTenantGroup
GetTenantGroup:
action: netbox.get.tenancy.tenant_groups name=<% ctx(customername) %>
next:
- when: <% succeeded() %>
publish:
- tgroupid : <% result().result.raw.results.id %>
do: CreateTenant
CreateTenant:
action: netbox.post.tenancy.tenants name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>, group=<% tgroupid %>
next:
- when: <% succeeded() %>

Running the action ‘netbox.get.tenancy.tenant_groups’ by itself via the UI, returns the following JSON output as a result:

{
  "stdout": "",
  "stderr": "",
  "exit_code": 0,
  "result": {
    "raw": {
      "count": 1,
      "next": null,
      "previous": null,
      "results": [
        {
          "id": 3,
          "name": "Some Company",
          "slug": "some-company",
          "parent": null,
          "description": "This is Some Company",
          "tenant_count": 1
        }
      ]
    }
  }
}

As you can see from the JSON output above, it returns the {“id”: 3} value that I’m trying to get assigned to the (tgroupid) variable that I have defined. Can someone please assist?

Also, I would like to understand how I can print the value of the variables at different places throughout a workflow to understand if/when they are changing. Similar to what you can do in any python script by printing the value at any point in a script.

I used the yaql evaluator and from what I can see this should have set your tgroupid to the value [3].

When you publish your are publishing variables to the context, so I think you just need to use ctx(“tgroupid”) or ctx().tgroupid to access your tgroupid variable you have set Instead of tgroupid directly like the example above uses.

amanda11,

I have made the adjustment/correction to calling the tgroupid variable; however, I have also narrowed down another part of my problem with getting this information out.

result: 
  errors:
  - message: 'TypeError: int() argument must be a string, a bytes-like object or a number, not ''list'''
    route: 0
    task_id: CreateTenant
    type: error
  output: null

As I have also had to work with this problem in Python, I always have to use the following in Python3 to process the returned list and get it into a dictionary.

tg_group_list = json.loads(json.dumps(netboxsvr.tenancy.get_tenant_groups(name=name),indent=4))

So now the question is, how can I get this information translated from a list to a dictionary in StackStorm (YAQL). (On a side note, having the result posted into the datastore and calling up st2kv.system results in the same issue, it gets stored as a list).

Any assistance would be greatly appreciated.

What is the dictionary that you want to have - what is the key/values needed? I wasn’t quite sure what dictionary you wanted to get out.

I normally use the yaql online evaluator to help me work out my YAQL expressions. So for example pasting in the JSON into http://yaqluator.com/, and then using $.result.raw.results.id for the example above it allows you to see what the result is.
So you can then experiment with the YAQL commands to see what gets you the result you want.

From this simple output, the only value required is the value of the “id”, so as you stated and in using the yaqluator, $.result.raw.results.id does return 3 in that tool. However, applying it in my workflow, always results in some sort of error:

input:

• customername

• customerslug

vars:

• tgroupid: null

tasks:

CreateTenantGroup:

action: netbox.post.tenancy.tenant_groups name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>

next:

  - when: <% succeeded() %>

    do: GetTenantGroup

GetTenantGroup:

action: netbox.get.tenancy.tenant_groups name=<% ctx(customername) %>

next:

  - when: <% succeeded() %>

    publish:

      - tgroupid : <% $.result.raw.results.id %>

    do: CreateTenant

CreateTenant:

action: netbox.post.tenancy.tenants name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>, group=<% ctx(tgroupid) %>

next:

  - when: <% succeeded() %>

When executing this, I always get some sort of error, most recent this:

parameters:
customername: Another Company
customerslug: another-company
status: failed
start_timestamp: Tue, 04 Aug 2020 22:53:06 UTC
end_timestamp: Tue, 04 Aug 2020 22:53:07 UTC
result:
errors:

• message: ‘YaqlEvaluationException: Unable to evaluate expression ‘’<% $.result.raw.results.id %>’’. NoFunctionRegisteredException: Unknown function “#property#result”’
  route: 0
  task_id: GetTenantGroup
  task_transition_id: CreateTenant__t0
  type: error
  output: null
  ±-------------------------±-----------------------±------------------±----------------------------------±------------------------------+
  | id | status | task | action | start_timestamp |
  ±-------------------------±-----------------------±------------------±----------------------------------±------------------------------+
  | 5f29e6d24fcba73398f42ff2 | succeeded (0s elapsed) | CreateTenantGroup | netbox.post.tenancy.tenant_groups | Tue, 04 Aug 2020 22:53:06 UTC |
  | 5f29e6d24fcba73398f42ff5 | succeeded (1s elapsed) | GetTenantGroup | netbox.get.tenancy.tenant_groups | Tue, 04 Aug 2020 22:53:06 UTC |
  ±-------------------------±-----------------------±------------------±----------------------------------±------------------------------+

I guess I don’t understand enough of the YAQL syntax. Problem is this is a very basic example, what happens when I need the full returned output in a dictionary where I can iterate through (the when function) over 100+ of this. But right now, I’d settle for getting this to work once

This is the other error I get when dealing with the data that is returned from any query:

message: ‘TypeError: int() argument must be a string, a bytes-like object or a number, not ‘‘list’’’

Note there is a difference between format in yaql evaluator and in the workflow.

In Yaql you use $.result.raw.resuls.id - but in the workflow you want to use result().result.raw.results.id

If you just want to get the first one out of the list, then you could use result().result.raw.results.id.first()

So I think following might work:
`

GetTenantGroup:
 action: netbox.get.tenancy.tenant_groups name=<% ctx(customername) %>

next:

  - when: <% succeeded() %>

    publish:

      - tgroupid : <% result().result.raw.results.id.first() %>

    do: CreateTenant
      action: netbox.post.tenancy.tenants name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>, group=<% ctx(tgroupid) %>

    next:

       - when: <% succeeded() %>

Yes, basically was using exactly what you have shown here. However, this still doesn’t return just the integer. According to both the yaqluator and via the error message from StackStorm.

result:
errors:

• message: ‘TypeError: int() argument must be a string, a bytes-like object or a number, not ‘‘list’’’
  route: 0
  task_id: CreateTenant
  type: error
  output: null

I’m struggling to get just the value to be returned. I have tried to use “dict($.result.raw.results)” to get the whole thing as a dictionary so I can call it through key:value. but this isn’t working either. Or at least, I haven’t figured out how to get this to work, the documentation for StackStorm shows the use of “.select” to call up values; however, the YAQL evaluator doesn’t recognize that as a valid function.

Could you post through the example and result when you used the .first() call to get the first item from the list, this shouldn’t have been returning you a list when you did that?

The YAQL evaluator does support select, for example you could use:
.result.raw.results.select(.id)
but that will give you again a list to iterate through again.

(NB when it’s a list you should be able to use the with-items to iterate through the ids - you wouldn’t need to have a dictionary).

But for now let’s get the first item out and working. Could you show me the results when you use

tgroupid : <% result().result.raw.results.id.first() %>

input:

• customername

• customerslug

vars:

• tgroupid: null

tasks:

CreateTenantGroup:

action: netbox.post.tenancy.tenant_groups name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>

next:

  - when: <% succeeded() %>

    do: GetTenantGroup

GetTenantGroup:

action: netbox.get.tenancy.tenant_groups name=<% ctx(customername) %>

next:

  - when: <% succeeded() %>

    publish:

      - tgroupid : <% result().result.raw.results.id.first() %>

    do: CreateTenant

CreateTenant:

action: netbox.post.tenancy.tenants name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>, group=<% ctx(tgroupid) %>

next:

  - when: <% succeeded() %>

This worked successfully. I wish I could just print the output or value stored in “tgroupid” as it changes from task to task. But, it apparently only returned the integer, otherwise it wouldn’t commit on NetBox.

This happens to be a simple example, so the bigger question for me will be a situation where I need multiple values from JSON output from NetBox, or as stated before multiple entries.

Glad that worked.

Do you want to call CreateTenant for each id?

In that case you could publish tgrouplist using result().result.raw.results.id (NB. the removal of first )

And then make CreateTenant use a with items where the list you give is the tgrouplist.

For example:

   CreateTenant:
      action: netbox.post.tenancy.tenants name=<% ctx(customername) %>, slug=<% ctx(customerslug) %>, group=<% item() %>
     with: 
         items: <% ctx().tgrouplist %>

This would then call the action once for every item in the tgrouplist - i.e. every id that we had.

–

If you want to build up a dictionary of say id to results, then you could use some YAQL similar to following to publish the results to a variable.

dict(result().result.raw.results.select([$.id,$]))

The selector selects the id from the results, as well as the whole entry for that results item. Then we wrap it in dict, and it builds up a dictionary where the key is the id, and the value is the full entry for that id. e.g. a result similar to

{
    "3": {
        "description": "This is Some Company",
        "id": 3,
        "name": "Some Company",
        "parent": null,
        "slug": "some-company",
        "tenant_count": 1
    },
    "4": {
        "description": "This is So2me Company",
        "id": 4,
        "name": "Some Co222mpany",
        "parent": null,
        "slug": "some-com2pany",
        "tenant_count": 2
    }
}

Amanda, thank you for all your help here, I wouldn’t be getting anywhere without your help.

But to be honest, I find the use of YAQL to be the reason to not use StackStorm at all. The documentation and examples are terrible and have to use the “yaqlator.com” site to just keep trying random things to get something out of the product is worthless.

Take the example you provided, I have to use YAQL to define a new dictionary because you can’t simply extract the one that is already there. Using

dict(result().result.raw.results.select([.id,]))

You do infact get the dictionary:

{
“3”: {
“description”: “This is Some Company”,
“id”: 3,
“name”: “Some Company”,
“parent”: null,
“slug”: “some-company”,
“tenant_count”: 1
}
}

Using the yaqlator, with the expression “$.get([id])” you get NULL returned. WTF. I’m assuming it’s because the “id” key is below that of the key “3”, but why can’t I just get a dictionary from "dict(result().result.raw.results) that looks like this:

{
“id”: 3,
“name”: “Some Company”,
“slug”: “some-company”,
“parent”: null,
“description”: “This is Some Company”,
“tenant_count”: 1
}

How do I get “live output”? Something that can tell me what the hell StackStorm is doing during the execution of the workflow.

Why does StackStorm make this shit so hard? Competing products like RunDeck (and others) make this part so much easier to debug and troubleshoot what the hell is going on during the workflow at execution to see every variable, command, timestamps, etc. With Stackstorm you get nothing. No documentation, examples in the software that are so basic and not real world (like adding 1+2). No examples on how to return data, log input and output of a workflow.

I’m not the expert on YAQL, it’s just my preference to use the YAQL evaluator for my own use. I think once you get used to it, it’s quite powerful and you can manipulate the values without having to write code.

In your example you are trying to convert an array into a dictionary, and so its not clear unless you tell it through the select what field you want the key to be - so there has to be some conversion.

I’m probably not the best person to answer how to debug seeing the values output through the flow. I can’t recall whether you get the information in the audit logs in /var/log/st2 or not. (I’ve not had a need - as I’ve just used the evaluator and the output result shown in the UI).

I’m sure someone more experienced can probably answer that bit better than me. It may be worth just asking that second bit about printing the values between actions in a separate forum post or on Slack, the community is really helpful so I’m sure we can find someone to answer that bit better than me!

PS To get the dictionary you wanted just use:

result().result.raw.results.first()

Remember that resuilt.raw.results is an array of dictionaries. So the select I did was turn that into a dictionary indexed on the id for each entry. But the example that you showed was just get the dictionary of the first entry - so the above will get you a dictionary that you mentioend.

Amanda. If I create the dictionary and define the key (id, as in the example above) I can no longer get the value of id. Even with the yaqlator. The .select([.id,$]) no longer works. So I don’t understand the logic of YAQL and what it’s doing. In Python, it’s quite easy to walk up and down the key:values of JSON or embedded JSON, but YAQL with the select, get doesn’t make any sense.

Unlike Python, where you can easily google almost anything you are trying to do, YAQL searches on google return less than 8 pages of results.

I’ll give this a shot, thx

I had a quick look at the forum for some advice on debugging. I did notice mention of another trick that I have used which is to put in core.echo task to see what variables are.
I am pretty sure the info is there in the logs (certainly would be if debug=True in [system] is put on in the st2.conf).