Do pack dependencies get resolved when building an HA docker image pack

This is not about python dependencies, it’s about st2 pack dependencies.

I ran the following docker build passing in a ‘stackstorm-ms’ pack which I think (not sure, but according to the contents of the pack.yml) defines dependencies on excel and powerpoint st2 packs.
If I were to do a regular st2 pack install, is it the case that any packs that are declared as dependencies in the pack.yml should be automatically resolved and installed? i.e. I should see excel and powerpoint packs installed in this case?

If so, shouldn’t it be the case that the st2-pack-install should also resolve dependencies (when building a docker HA image)?

In the following attempt to load custom packs I don’t see any mention of excel or powerpoint in this image (should I?)

chatops         core            default         linux           microsoft_test  packs

Sending build context to Docker daemon  4.608kB
Step 1/3 : FROM stackstorm/st2packs:builder AS builder
# Executing 2 build triggers
 ---> Using cache
 ---> Running in e36e2043b5c8
Removing intermediate container e36e2043b5c8
 ---> a62b82d23bc8
Step 2/3 : RUN /opt/stackstorm/st2/bin/st2-pack-install ${PACKS}
 ---> Running in c90915fa197d
2020-01-28 11:28:47,101 INFO [-] Installing pack "https://github.com/StackStorm/stackstorm-ms.git"
2020-01-28 11:28:51,997 INFO [-] Successfully installed pack "microsoft_test"
2020-01-28 11:28:51,997 INFO [-] Setting up virtualenv for pack "microsoft_test"
2020-01-28 11:28:51,999 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/microsoft_test" doesn't exist
2020-01-28 11:28:54,079 INFO [-] Successfully set up virtualenv for pack "microsoft_test"
Removing intermediate container c90915fa197d
 ---> 89063e628a5e
Step 3/3 : FROM stackstorm/st2packs:runtime
# Executing 3 build triggers
 ---> Using cache
 ---> bc098cda997a
Successfully built bc098cda997a
Successfully tagged st2packs:latest

Assuming there’s no dependency resolution. Is there a way I can evaluate the set of packs I require while online? (I can then pass this directly to the image). I can write something to check the dependencies section and resolve it manually, but i’d rather not reinvent the wheel, if there’s something already available or if there’s a way to make use of the existing st2 pack resolving mechanism i’d rather use that.

Based on PR you contributed Upgrading st2:3.0dev to st2:3.2dev by danielburrell · Pull Request #17 · StackStorm/st2packs-dockerfiles · GitHub stackstorm/st2packs Docker image is now relying on st2 3.2dev which includes this updated functionality.

Not sure if new pack dependency feature was tested with the minimal docker st2-pack-install script. Please let me know if that works as expected or not.

Thanks

It doesn’t seem to work as expected. It would be good to know what the basis for resolving dependencies is.

To confirm is the following true:

• the pack.yaml drives the content of any index file.
• the index file is the authority on dependencies
• the ‘version’ pulled in is by tag, and so the version in the pack.yaml that is tagged in git as 0.0.1 must be set to 0.0.1 also. (st2 will blindly pull the tag that matches the version in git).

Dockerfile is here (uses a custom mst2.conf which is the same as the original except the [content].index is set to my custom index. Index is shown below.

FROM stackstorm/st2packs:builder AS builder
RUN /opt/stackstorm/st2/bin/st2-pack-install -vv --config-file /mst2.conf ${PACKS}
FROM stackstorm/st2packs:runtime

Here’s the images that are in my cache from that build (they match the digests for 3.2dev)

stackstorm/st2packs                                                          runtime             sha256:dd340841edc2ce770a6c0fbac22e561319e65194f3a0648d27777dba76f6999e   d6946af13b61        5 hours ago         4.41MB
stackstorm/st2packs                                                          builder             sha256:a5c4641adff2e41ef72c23c793cba8962d2e2a594053b1657b6d56e449c7b4c2   824cd0e9de35        6 hours ago         540MB

Here is the execution. Note it starts immediately with projecta (isn’t it supposed to start with the dependencies?)

#packs
Step 12/13 : RUN /opt/stackstorm/st2/bin/st2-pack-install -vv --config-file /mst2.conf "${PACKS}"
 ---> Running in ac8020ae36ae
2020-02-11 16:19:14,841 INFO [-] Installing pack "projecta=0.0.1"
2020-02-11 16:19:21,749 INFO [-] Successfully installed pack "projecta"
2020-02-11 16:19:21,750 INFO [-] Setting up virtualenv for pack "projecta"
2020-02-11 16:19:21,751 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/projecta" doesn't exist
2020-02-11 16:19:24,661 INFO [-] Successfully set up virtualenv for pack "projecta"

I have provided a custom index here

{
  "metadata" : {
    "generated_ts" : 1581371761034,
    "hash" : "beb6c5c7de3dabf56b593c0bfa7f882822f9e3e8847a1b9f6a311c2dab68d931"
  },
  "packs" : {
    "projecta" : {
      "author" : "Only Me",
      "content" : {
        "actions" : {
          "count" : 0,
          "resources" : [ ]
        },
        "tests" : {
          "count" : 0,
          "resources" : [ ]
        }
      },
      "description" : "Company MaskedA automation",
      "email" : "[email protected]",
      "keywords" : [ "projecta" ],
      "name" : "MaskedA",
      "ref" : "projecta",
      "repo_url" : "[email protected]:superproject/automation-packs/projecta.git",
      "version" : "0.0.1",
      "versions" : [ "0.0.1" ],
      "dependencies" : [ "projectb=0.0.1" ]
    },
    "projectb" : {
      "author" : "Only Me",
      "content" : {
        "actions" : {
          "count" : 0,
          "resources" : [ ]
        },
        "tests" : {
          "count" : 0,
          "resources" : [ ]
        }
      },
      "description" : "Company Masked Pack",
      "email" : "[email protected]",
      "keywords" : [ "projectb" ],
      "name" : "Masked",
      "ref" : "projectb",
      "repo_url" : "[email protected]:superproject/integration-packs/prefix-projectb.git",
      "version" : "0.0.1",
      "versions" : [ "0.0.1" ]
    }
  }
}

The projecta and projectb both consist of single pack.yaml file. in projecta, the file also expresses a dependency on projectb. both are versioned at 0.0.1 and both have a tag at 0.0.1

Expected behaviour would be that on completion, both projecta and projectb are installed having only specified projecta needs to be installed since projecta depends on projectb in the pack.yaml

Here’s projecta pack.yaml

---
ref: projecta
name: ProjectA
description: Company ProjectA automation
keywords:
  - projecta
version: 0.0.1
author: Its Me
email: [email protected]
dependencies:
  - projectb=0.0.1

To simplify this case and instead of creating your own StackStorm Exchange index, try to install the following pack https://github.com/StackStorm/stackstorm-ms.git. It has 2 dependencies in its pack metadata.

If it’s not working as expected, please report a bug in stackstorm/st2 repository. It could be an oversight to take into account st2-pack-install during the pack dependencies feature development.
Thanks!

Thanks for providing a pack with a dependency. I couldn’t find one in the main index.

Yep, I don’t see any of the dependencies being installed here

Step 9/10 : RUN /opt/stackstorm/st2/bin/st2-pack-install -vv https://github.com/StackStorm/stackstorm-ms

 ---> Running in 671389dc775e
2020-02-13 17:44:53,902 INFO [-] Installing pack "https://github.com/StackStorm/stackstorm-ms"
2020-02-13 17:44:53,903 DEBUG [-] Popen(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'], cwd=/opt/stackstorm, universal_newlines=True, shell=None)
2020-02-13 17:44:55,509 DEBUG [-] Cmd(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'])'s unused stdout: 
2020-02-13 17:44:55,518 DEBUG [-] Popen(['git', 'cat-file', '--batch-check'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,256 DEBUG [-] Popen(['git', 'branch', '-a', '--contains', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,906 DEBUG [-] Popen(['git', 'checkout', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:57,545 DEBUG [-] Popen(['git', 'branch', '-f', 'master', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,292 DEBUG [-] Popen(['git', 'checkout', 'master'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,940 DEBUG [-] Moving pack from /root/6622c8eb7209250c62980483f40577e2 to /opt/stackstorm/packs.
2020-02-13 17:44:58,942 DEBUG [-] Changing owner group of "/opt/stackstorm/packs/microsoft_test" directory to st2packs
2020-02-13 17:44:58,942 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:44:58,942 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:44:58,942 DEBUG [-] Creating subprocess.
2020-02-13 17:44:58,948 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:44:58,949 DEBUG [-] Attaching to process.
2020-02-13 17:44:58,949 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:44:58,949 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:44:58,963 DEBUG [-] No timeout.
2020-02-13 17:44:58,963 DEBUG [-] Returning.
2020-02-13 17:44:58,966 INFO [-] Successfully installed pack "microsoft_test"
2020-02-13 17:44:58,966 INFO [-] Setting up virtualenv for pack "microsoft_test"
2020-02-13 17:44:58,968 DEBUG [-] Setting up virtualenv for pack "microsoft_test" (/opt/stackstorm/packs/microsoft_test)
2020-02-13 17:44:58,968 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/microsoft_test" doesn't exist
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv for pack "microsoft_test" in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv in "/opt/stackstorm/virtualenvs/microsoft_test" using Python binary "/opt/stackstorm/st2/bin/python"
2020-02-13 17:44:58,969 DEBUG [-] Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --no-download /opt/stackstorm/virtualenvs/microsoft_test" to create virtualenv.
2020-02-13 17:45:01,563 DEBUG [-] Installing base requirements
2020-02-13 17:45:01,563 DEBUG [-] Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/microsoft_test/bin/pip install six>=1.9.0,<2.0.
2020-02-13 17:45:02,018 DEBUG [-] No pack specific requirements found
2020-02-13 17:45:02,018 DEBUG [-] Changing owner group of "/opt/stackstorm/virtualenvs/microsoft_test" directory to st2packs
2020-02-13 17:45:02,018 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:45:02,019 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:45:02,019 DEBUG [-] Creating subprocess.
2020-02-13 17:45:02,022 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:45:02,022 DEBUG [-] Attaching to process.
2020-02-13 17:45:02,022 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:45:02,022 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:45:02,044 DEBUG [-] No timeout.
2020-02-13 17:45:02,044 DEBUG [-] Returning.
2020-02-13 17:45:02,044 DEBUG [-] Virtualenv for pack "microsoft_test" successfully created in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:45:02,044 INFO [-] Successfully set up virtualenv for pack "microsoft_test"

Bug raised