Define a workflow in Python

Hi guys,
I’m new to StackStorm and currently I’m exploring it’s abilities.

It seem very cool, but the Orquesta-based workflow it’s not so good for me.
In my use case I must manage very complex workflows, and I don’t want to go crazy by defining them with the YAML DSL provided by Orquesta.

At the same time, I would like to take advantage of the public packs available in the Exchange. So, essentially, I’d like to define my workflows using Python actions, invoking other actions (for instance, to query Active Directory, Oracle database, and so on) from the main one.

I have just seen the post Invoke an action from a Python action script, but the proposed solution requires a continuous polling of StackStorm, in order to check the completion of each action. So I think that this solution is very inefficient when trying to invoke many actions, and when you have many instances of your main action running.

Maybe, after two years, is there a better solution to do it?

Triggering one action from another and trying to workflow in python actions is an anti-pattern in the StackStorm world. These automations are best built and managed by re-usable blocks like Packs, Actions, Sensors and Rules separating the concerns.

If Orquesta looks too complex to you, try ActionChain — StackStorm 3.2.0 documentation which is linear and pretty straightforward. It also the fastest workflow engine because of its simplicity.

What is the example use case/logic you’d like to explore?

Let’s suppose, for demonstration purposes only, a workflow like to the following (in pseudocode):

WhenSomeEventOccours(params) {
  try {
    account = activeDirectoryLookupUser(params.uid)
    if account not found {
      publishEvent("RequirementNotSatisfied")
      abort()
    }
    assert(account is enabled)
    assert(activeDirectoryLookupGroup("root/{params.foo}/Groups/{params.bar}") is found)
    assert(httpGet("https://example.org/{params.zoo}") is 200)
    if params.alternativeUid {
      account = activeDirectoryLookupUser(params.alternativeUid)
      if account not found {
        account = activeDirectoryCreateAccount(params...)
        sendMail(account credentials)
      }
    }
    activeDirectoryAddUserToGroup(account, "OU/{params.foo}/Groups/{params.bar}")
    if (params.sync) {
      httpPost("https://example.org/{params.zoo}/member", account)
      rows = doSqlQuery("select * from ...")
      for each row in rows {
        httpPost("https://example.org/{params.zoo}/apply", row)
        httpPost("https://example.org/{params.zoo}/row/{row.id}/restrict", account)
      }
    }
    publishEvent("WorkflowCompleted", account, params)
  } catch (error) {
    publishEvent("WorkflowFailed", params, error)
    // then some event listener could send a notification to report the failure
  }
}

I think that every function such as activeDirectoryLookupUser, httpGet, doSqlQuery, etc. could come from an existent action pack, but the glue is not so trivial…

Nice!

Yes, exactly.
You want to split every logical step into a dedicated pack/action. And replace WhenSomeEventOccours with the StackStorm Rule (Rules — StackStorm 3.2.0 documentation).

What you have is pretty much a linear chain of actions. I think it’s OK to keep some of the custom logic in one place as you start. However as this code grows it becomes harder to manage, maintain, improve, re-use it.

In StackStorm world, Actions = Code and Workflows = yaml . They complement each other.

Yes, you can do some linear multi-step work and put everything in one Javascript/Python/any_language code Action and that’s no problem with that, if that works to start with something.

However to achieve something more useful/complex which includes nested/branched logic Workflows are inevitable.

Examples:

Can you handle that nicely with just Python/Javascript code placed in one Action?

Workflows are known to have 40 control patterns: http://www.workflowpatterns.com/patterns/control/ if not more. It becomes hard to write and maintain code which needs more advanced stuff. This is done with workflows:

• run several tasks in parallel
• merge their results
• split
• join
• filter
• conditions
• loops
• synchornize
• retry
• crash recovery, etc.
• logging
• even wait for confirmation from human when needed (Inquiries — StackStorm 3.2.0 documentation)

That’s where workflows help with its own DSL, structure, maintainability, readability, organization and clarity, while common actions are re-usable and shareable bricks. This gives a good separation of concerns and abstraction. StackStorm provides a framework to also observe the flow via UI like that, debug, manage individual steps and so on.

Think about it ^^ from the perspective of Monolyth vs Microservices approach. Pretty similar strategy here with the Workflows. Hope that helps.

Thank you @armab for yours replies!

I’m sure it’s possible to do it using Orquesta, but it could be overkill for me (at least at the beginning), and sure, using Rules I can bind events to custom scripts in any language, but my initial question is a little bit different.

Let’s suppose that I write a Python script equivalent to the previous example.
The invocation of the activeDirectoryLookupUser pseudo-function could be translated in the execution of the action get_ad_user from the pack stackstorm-activedirectory.
So I would like to use it in my action, maybe through a high level API provided by StackStorm, instead of adopting some kind of library.

You said that it is an anti-pattern, but in my opinion it is a valid option when you want all the power of a programming language, along with the flexibility of StackStorm.

Sure, I think you’ve got the workaround described in Invoke an action from a Python action script for a quick start.

@m4dcoder What are your opinions about workflows as a python code in StackStorm actions?

If orchestrating a workflow using different actions in packs, the supported method is to write in workflows such as orquesta or action chain (mistral is about to be deprecated). Workflow also allows you to break down complex series of operations into multiple steps and take advantage of the engine’s capability to run it asynchronously, scaled horizontally across multiple action runners, and other operations such as pause/resume/cancel/rerun/etc. This is what StackStorm is designed to do and this is tested.

If you want the simplicity of being triggered by an event and then just run a script, you can always just code the series of operations that you want to do in python or shell scripts. This is also what StackStorm is designed and tested for. Especially if you are naturally a coder, I can understand why writing workflows in YAML is overkill. You can copy the codes from other packs and assembly them in your pack, create modules for AD and SQL and have you python action calls those modules. There’s nothing wrong with that.

There’s also a balance between writing every single line of code in workflow vs. breaking down workflow into high level logical steps and then writing the low level operations in python actions.

I don’t really recommend calling StackStorm actions from within another StackStorm action. It is possible. This just makes the action duplicate features which the StackStorm platform is already doing and now you have to do the testing and support the complexity of that action.