We have a workflow with a concurrency of 5. A few months ago, it hummed right along and our processing was done in a few hours. But this seems to be degrading over time.
psla_ingest_new_cves kicks off five tasks at the same time, but then it sits and waits forever before starting the next batch of 5, which don’t all start at exactly the same time.
I’m wondering why this started happening. Is the on-success and on-error run time included in this time? Logs are difficult to read. I’ve recently taken over this project and am pretty new to stackstorm, and the mistral documentation has not been very helpful. While the st2 cli is pretty useful for finding and retrieving execution output. mistral and mistral documentation is less helpful.
Any tips on what this might be or how to debug it would be much appreciated.
psla_ingest_new_cves:
with-items: service in {{ _.all_services_list }}
concurrency: 5
action: patch_orchestrator.update_patch_sla_db
input:
service: "{{ _.service }}"
timeout: 43200
use_cache: true
publish:
results_msgs: "{{ task('psla_ingest_new_cves').result | map(attribute='result') | map(attribute='msg') | list }}"
on-success:
- notify: "{{ _.notify_on_success }}"
on-error:
- notify_and_fail: "{{ _.notify_on_failure }}"
+--------------------------+--------------------------+-----------------------+-----------------------+--------------------+
| id | status | task | action | start_timestamp |
+--------------------------+--------------------------+-----------------------+-----------------------+--------------------+
| 60265fb1e7c95c4f790b5502 | succeeded | create_cache | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | (3816s elapsed) | | date_patch_sla_create | 03:00:01 PST |
| | | | _cache | |
| 60266e9ae7c95c4f790b5795 | succeeded | psla_update_pending_c | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | (6749s elapsed) | ves | date_patch_sla_close_ | 04:03:38 PST |
| | | | cves | |
| 602688f9e7c95c4f790b655e | succeeded (7s elapsed) | get_all_services | patch_orchestrator.se | Fri, 12 Feb 2021 |
| | | | rvice_metadata | 05:56:09 PST |
| 60268902e7c95c4f790b6560 | succeeded (95s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 05:56:18 PST |
| 60268902e7c95c4f790b6562 | succeeded (35s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 05:56:18 PST |
| 60268902e7c95c4f790b6564 | succeeded (276s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 05:56:18 PST |
| 60268903e7c95c4f790b6567 | succeeded (34s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 05:56:18 PST |
| 60268903e7c95c4f790b6568 | succeeded (38s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 05:56:18 PST |
| 6026a696e7c95c4f790b659f | succeeded (33s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 08:02:30 PST |
| 6026a697e7c95c4f790b65a1 | succeeded (31s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 08:02:31 PST |
| 6026a69ae7c95c4f790b65a3 | succeeded (64s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 08:02:34 PST |
| 6026a6d5e7c95c4f790b65a5 | succeeded (214s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 08:03:33 PST |
| 6026a786e7c95c4f790b65a7 | succeeded (48s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 08:06:30 PST |
| 6026b8cde7c95c4f790b667a | succeeded (430s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 09:20:13 PST |
| 6026c338e7c95c67204f1290 | succeeded (61s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 10:04:40 PST |
| 6026c338e7c95c67204f1292 | succeeded (54s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |
| | | | date_patch_sla_db | 10:04:40 PST |
| 6026c35ce7c95c67204f1294 | succeeded (51s elapsed) | psla_ingest_new_cves | patch_orchestrator.up | Fri, 12 Feb 2021 |