StackStorm v3.2dev pre-release Testing

[arm4b]

We're ready to prepare the StackStorm v3.2 release and starting pre-release testing.

This will be the first StackStorm release under the Linux Foundation with new Open Source Governance and Maintainers team. We now rely on our Community more than ever and invite everyone to help with the pre-release testing and development/contribution.

Release Process Preparation

Per Release Management Schedule @armab is the Release Manager and @punkrokk Assisting for v3.2. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels.
The first step is pre-release manual user-acceptance testing for v3.2dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough.
There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.2dev staging packages, try random things in random environments (different OS) and report any regressions found comparing to v3.1:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

Extra points for PR hotfixes and adding new or missing test cases.

Major changes

Important changes which are highly encouraged to try and test.

• New CentOS8/RHEL8 support with Python 3.6 and MongoDB 4.0:
  https://docs.stackstorm.com/latest/install/rhel8.html
• New st2 pack install with dependencies:
  https://docs.stackstorm.com/latest/packs.html#pack-dependencies
• New Orquestra re-run:
  https://docs.stackstorm.com/latest/orquesta/operations.html#re-running-workflow-execution
  https://docs.stackstorm.com/latest/orquesta/operations.html#re-running-workflow-execution-from-task-s
• New Orquestra task retry in workflow definition:
  https://docs.stackstorm.com/latest/orquesta/languages/orquesta.html#task-retry-model
• New Orquestra continue, noop, fail keywords in workflow definition:
  https://docs.stackstorm.com/latest/orquesta/languages/orquesta.html#engine-commands
• ChatOps functionality with Slack, Microsoft Teams, Cisco Spark (Webex) and Mattermost might be affected due to refactoring, fixes and major updates.
• Test v3.1 -> v3.2 upgrade. Upgrade Notes:
  https://docs.stackstorm.com/latest/upgrade_notes.html#st2-v3-2
• Run st2-self-check
  https://docs.stackstorm.com/latest/troubleshooting/self_verification.html

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

• Add support for blacklisting / whitelisting hosts to the HTTP runner by adding new
  url_hosts_blacklist and url_hosts_whitelist runner attribute. (new feature)
  Add support for blacklisting hosts to the HTTP runner st2#4757
• Add user parameter to re_run method of st2client. Add user field to re_run method st2#4785
• Install pack dependencies automatically. Pack dependency management st2#4769
• Add support for immutable_parameters on Action Aliases. This feature allows default
  parameters to be supplied to the action on every execution of the alias. Add support for immutable_parameters in ChatOps action aliases st2#4786
• Add get_entrypoint() method to ActionResourceManager attribute of st2client.
  GET entrypoint of a workflow from st2client st2#4791
• Add support for orquesta task retry. (new feature)
• Add config option scheduler.execution_scheduling_timeout_threshold_min to better control the cleanup of scheduled actions that were orphaned. Make scheduling timeout constant to be a configurable option st2#4886

Changed

• Install pack with the latest tag version if it exists when branch is not specialized.
  (improvement) Install pack with the latest tag st2#4743

• Implement "continue" engine command to orquesta workflow. (improvement) Orquesta workflows have incorrect status st2#4740

• Update various internal dependencies to latest stable versions (apscheduler, eventlet,
  kombu, amqp, pyyaml, mongoengine, python-gnupg, paramiko, tooz, webob, bcrypt).

  Latest version of mongoengine should show some performance improvements (5-20%) when
  writing very large executions (executions with large results) to the database. Update various Python dependencies to latest stable versions st2#4767

• Improved development instructions in requirements.txt and dist_utils.py comment headers
  (improvement) Fix dependency conflicts st2#4774

• Add new actionrunner.stream_output_buffer_size config option and default it to -1
  (previously default value was 0). This should result in a better performance and smaller
  CPU utilization for Python runner actions which produce a lot of output.
  (improvement)

  Reported and contributed by Joshua Meyer (@jdmeyer3) fixing subprocess to use system buffer instead of being unbuffered st2#4803

• Add new action_runner.pip_opts st2.conf config option which allows user to specify a list
  of command line option which are passed to pip install command when installing pack
  dependencies into a pack specific virtual environment. Don't call eventlet directly inside the st2reactor code st2#4792

• Refactor how orquesta handles individual item result for with items task. Before the fix,
  when there are a lot of items and/or result size for each item is huge, there is a negative
  performance impact on write to the database when recording the conductor state. (improvement)

• Remove automatic rendering of workflow output when updating task state for orquesta workflows.
  This caused workflow output to render incorrectly in certain use case. The render_workflow_output
  function must be called separately. (improvement)

• Update various internal dependencies to latest stable versions (cryptography, jinja2, requests,
  apscheduler, eventlet, amqp, kombu, semver, six) Update pip dependencies, fix conflicting dependency version, ensure there are no conflicting dependencies st2#4819 (improvement)

• Improve MongoDB connection timeout related code. Connection and server selection timeout is now
  set to 3 seconds. Previously a default value of 30 seconds was used which means that for many
  connection related errors, our code would first wait for this timeout to be reached (30 seconds)
  before returning error to the end user. Fix SSL support for MongoDB and RabbitMQ under Python 3.x st2#4834

• Upgrade pymongo to the latest stable version (3.10.0.). Update pymongo to the latest version st2#4835 (improvement)

• Updated Paramiko to v2.7.1 to support new PEM ECDSA key formats Fixes: #4890 to support new ssh key types st2#4901 (improvement)

• Remove .scrutinizer.yml config file. No longer used.

• Convert escaped dict and dynamic fields in workflow db models to normal dict and dynamic fields.
  (performnce improvement)

• Add support for PEP 508 <https://www.python.org/dev/peps/pep-0508/stackstorm/st2#environment-markers>_
  environment markers in generated requirements.txt files. (improvement) Various requirements fixups st2#4895

• Use pip-compile from pip-tools instead of pip-conflict-checker (improvement) Fix futurist version conflict st2#4896

• Refactor how inbound criteria for join task in orquesta workflow is evaluated to count by
  task completion instead of task transition. (improvement)

• The workflow engine orquesta is updated to v1.1.0 for the st2 v3.2 release. The version upgrade
  contains various new features and bug fixes. Please review the release notes for the full list of
  changes at https://github.com/StackStorm/orquesta/releases/tag/v1.1.0 and the st2 upgrade notes
  for potential impact. (improvement)

Fixed

• Fix the action query when filtering tags. The old implementation returned actions which have the
  provided name as action name and not as tag name. (bug fix) Tags filtering is not returning the expected result through the API st2#4828

  Reported by @angrydeveloper and contributed by Marcel Weinberg (@winem)

• Fix the passing of arrays to shell scripts where the arrays where not detected as such by the
  st2 action_db utility. This caused arrays to be passed as Python lists serialized into a string.

  Reported by @kingsleyadam Shell script action not serializing array properly. st2#4804 and contributed by Marcel Weinberg (@winem) #4804 Handle action parameters of type array correctly st2#4861

• Fix ssh zombies when using ProxyCommand from ssh config Close proxycommand socket when its not needed anymore and resolve ssh zombies st2#4881 [Eric Edgar]

• Fix rbac with execution view where the rbac is unable to verify the pack or uid of the execution
  because it was not returned from the action execution db. This would result in an internal server
  error when trying to view the results of a single execution.
  Contributed by Joshua Meyer (@jdmeyer3) With enterprise LDAP, retrieve single execution returns 500 st2#4758

• Fixed logging middleware to output a content_length of 0 instead of Infinity
  when the type of data being returned is not supported. Previously, when the value was
  set to Infinity this would result in invalid JSON being output into structured
  logs. (bug fix) Infinity values in log messages cause issues when parsing JSON st2#4722

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Fix the workflow execution cancelation to proceed even if the workflow execution is not found or
  completed. (bug fix) Orquesta workflow execution is stuck with canceling status st2#4735

• Added better error handling to contrib/linux/actions/dig.py to inform if dig is not installed.
  Contributed by JP Bourget (@punkrokk Syncurity) Update dig.py error handling st2#4732

• Update dist_utils module which is bundled with st2client and other Python packages so it
  doesn't depend on internal pip API and so it works with latest pip version. (bug fix) Fix dist_utils.py so it doesn't rely on internal pip API st2#4750

• Fix dependency conflicts in pack CI runs: downgrade requests dependency back to 0.21.0, update
  internal dependencies and test expectations (amqp, pyyaml, prance, six) (bugfix) Fix dependency conflicts st2#4774

• Fix secrets masking in action parameters section defined inside the rule when using
  GET /v1/rules and GET /v1/rules/<ref> API endpoint. (bug fix) Added mask of rule action secret parameters st2#4788 /v1/rules API endpoint action parameters secrets masking (additional changes on top of #4788) st2#4807

  Contributed by @Nicodemos305 and @jeansfelix

• Fix a bug with authentication API endpoint (POST /auth/v1/tokens) returning internal
  server error when running under gunicorn and whenauth.api_url config option was not set.
  (bug fix) Fix a bug with authentication API endpoint returning internal server error under gunicorn if auth.api_url config option was not set st2#4809

  Reported by @guzzijones

• Fixed st2 execution get and st2 run not printing the action.ref for non-workflow
  actions. (bug fix) st2 execution get doesn't print action.ref for non-workflow actions st2#4739

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Update st2 execution get command to always include context.user, start_timestamp and
  end_timestamp attributes. (improvement) st2 execution get doesn't print action.ref for non-workflow actions st2#4739

• Fixed core.sendmail base64 encoding of longer subject lines (bug fix) core.sendmail action UTF-8/base64 encoding doesn't handle longer subject lines properly st2#4795

  Contributed by @stevemuskiewicz and @guzzijones

• Update all the various rule criteria comparison operators which also work with strings (equals,
  icontains, nequals, etc.) to work correctly on Python 3 deployments if one of the operators is
  of a type bytes and the other is of a type unicode / string. (bug fix) Update rule comparison operators to work with mixed operator types (bytes and strings) st2#4831

• Fix SSL connection support for MongoDB and RabbitMQ which wouldn't work under Python 3 and would
  result in cryptic "maximum recursion depth exceeded while calling a Python object" error on
  connection failure.

  NOTE: This issue only affected installations using Python 3. (bug fix) Connection to mongo db over SSL doesn't work st2#4832 Fix SSL support for MongoDB and RabbitMQ under Python 3.x st2#4834

  Reported by @alexku7.

• Fix the amqp connection setup for WorkflowExecutionHandler to pass SSL params. (bug fix) Fix the amqp connection for WorkflowExecutionHandler to pass the SSL params st2#4845

  Contributed by Tatsuma Matsuki (@mtatsuma)

• Fix dependency conflicts by updating requests (2.23.0) and gitpython (2.1.15). Fix master builds st2#4869

• Fix orquesta syntax error for with items task where action is misindented or missing. (bug fix)
  PR Fix schema of with items task to not allow additional properties orquesta#195.

• Fix orquesta yaql/jinja vars extraction to ignore methods of base ctx() dict. (bug fix)
  PR Fix the yaql/jinja vars extraction to ignore methods of base ctx() dict orquesta#196. Fixes Orquesta workflow ctx().get() throws a YAQL error st2#4866.

• Fix parsing of array of dicts in YAQL functions. Fix regression in YAQL/Jinja conversion
  functions as a result of the change. (bug fix) PR Fixed a problem not to be able to parse input value which is type of dict in array in some YAQL operators. orquesta#191.

  Contributed by Hiroyasu Ohyama (@userlocalhost)

Removed

• Removed Ubuntu 14.04 from test matrix Don't test or build Ubuntu 14 packages st2#4897

Full list of changes: https://github.com/StackStorm/st2/blob/master/CHANGELOG.rst

orquesta

Added

• Add feature to retry task execution on given condition with delay in between retries and
  max number of retries. (new feature)
  Contributed by Hiroyasu Ohyama (@userlocalhost) and Nick Maludy (@nmaludy)
• Add support to rerun completed workflow (failed or succeeded) from given task(s). (new feature)

Changed

• Replace "noop" with "continue" when "do" is not specified. The new "continue" command
  will not alter the previous task state and will continue to conduct the workflow
  execution. Orquesta workflows have incorrect status st2#4740 (improvement)
• Refactor conductor to not store each item result in task state. If there are a lot of items
  and/or result size is huge per item, then there will be a performance impact on database
  write operations when recording the conductor state. (improvement)
• Use ujson to deepcopy dict(s) for faster performance. (improvement)
• Refactor how inbound criteria for join task is evaluated to count by task completion
  instead of task transition. (improvement)

Fixed

• Fix conducting of cycle with a fork. Fixes Tasks in a fork from a cycle do not execute orquesta#169 (bug fix)
• Fix request_workflow_status to ignore certain status change errors such as pausing a workflow
  that is already pausing and canceling a workflow that is already canceling. (bug fix)
• Remove rendering of workflow output automatically when updating task state. This caused
  workflow output to render incorrectly in certain use case. The render_workflow_output function
  must be called separately. (bug fix)
• When inspecting custom YAQL/Jinja function to see if there is a context arg, use getargspec
  for py2 and getfullargspec for py3. (bug fix)
• Check syntax on with items task to ensure action is indented correctly. Fixes Workflow stuck in running state orquesta#184 (bug fix)
• Fix variable inspection where ctx().get() method calls are identified as errors.
  Fixes Orquesta workflow ctx().get() throws a YAQL error st2#4866 (bug fix)
• Fix a problem of TypeError orccuring when a list (or dict) value that contains unhashable typed
  value (list or dict) is passed in some YAQL operators (e.g. distinct()). Fixes YAQL methods can't handle an array of dict value at 'items' of 'with' parameter orquesta#176 (bug fix)
  Contributed by Hiroyasu Ohyama (@userlocalhost)

Fore more info see https://github.com/StackStorm/orquesta/blob/master/CHANGELOG.rst

st2chatops

• Massive refactoring for hubot-stackstorm st2chatops relies on. Random ChatOps bugs could be fixed as well as regressions appear.
• Update to latest fixed Microsoft Teams Hubot Adapter upgrade hubot-botframework to commit #01d5be9 st2chatops#141 Update st2chatops to use fixed Microsoft Teams and Cisco Spark adapters st2chatops#148
• Update to latest fixed Cisco Spark (Webex) Hubot Adapter Chatops does not seem to be working for cisco spark [webex teams] hubot-stackstorm#198 Fix issue with webex adapter hubot-stackstorm#200 Update st2chatops to use fixed Microsoft Teams and Cisco Spark adapters st2chatops#148
• Fix for Mattermost Adapter Fix Mattermost adapter hubot-stackstorm#192

st2web

• No changes.

Warning!
st2web UI needs a contributor/maintainer! Security alerts due to outdated dependencies, as well as tech debt, issues and number of unreviewed PRs is all high.

---

This could be huge. With almost 1 year since v3.1 st2 release codebase accumulated so many new features, changes and fixes.

Please report findings here and bugs/regressions in respective repositories.
Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

[arm4b]

@StackStorm/maintainers Even if you didn't find any bugs, please report back in this discussion which OS/environment you've used and which features/items you've tried or verified. This will help to get an overall idea about OS/distro testing distribution.
I may assign OS/flavors to different team members as we did before.

[amanda11]

Very brief test done on centos8. Installed using single line install which went fine.

Simple chatops configured with slack for the aws ec2 get alias (note: stackstorm_aws 1.2.3 pack used, as 1.3.1 had issues - which I'll report separately - but not to do with stackstorm version).
Simple orquesta workflow moved over successfully from stackstorm 3.1 install, which used the aws_boto3 and slack packs.

I did fall foul to the change where it installs the latest tagged pack rather than latest version of the code, on page: https://docs.stackstorm.com/latest/packs.html it states that it installs the latest commit of a pack rather than the tagged, but looking at the change log then that seems to be a known change.
Docs state "By default, the latest commit to a pack repository will be installed, but you can specify a particular version, branch, tag, or even a commit hash. Just use =:" yet, it seems to install last tag. If this is by design as per "Install pack with the latest tag version if it exists when branch is not specialized. (improvement) #4743" - then does documentation need updating?

[arm4b]

@amanda11 Awesome, thank you very much for testing and feedback!

Yes indeed, in 3.2dev the behavior of pack install has changed which now installs latest "tag/version" instead of latest commit from master.
That's a good catch! 👍

@amanda11 Documentation indeed needs updating. How about submitting a small PR to fix that?
https://github.com/StackStorm/st2docs/blob/master/docs/source/packs.rst

[amanda11]

@armab I'll submit a PR for the doc change. thanks.

[amanda11]

@armab I'll submit a PR for the doc change. thanks.

StackStorm/st2docs#973 raised. I also verified the behaviour when you had a local repo that it was working, and what it picked up if you used file:/// towards a local git repo. That seems to still use the latest commit when using a file:/// to a local repo - so I left that section stating latest commit. So I literally just changed one word in the docs in the end.

[arm4b]

FYI for someone testing 3.1 -> 3.2 upgrade scenarios here is how to install 3.1dev from unstable packages first:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- \
  --unstable --version=3.1dev \
  --user=st2admin --password=Ch@ngeMe

Then follow the packages upgrade procedure https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure to get v3.1dev->v3.2dev upgraded.

[arm4b]

Installation with the following instructions fails on EL6 and EL7:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s --   --unstable --version=3.1dev   --user=st2admin --password=Ch@ngeMe

repoquery -y flag which was added before is non-existent for EL6.
Update: -y is non-existent for EL7 as well.

Reported as StackStorm/st2-packages#641 (Fixed now)

[amanda11]

Further positive feedback on testing. Loving the immutable parameters on the aliases. Needed them for some aliases I needed for AWS reporting using the aws_boto3 pack. Integrated those successfully with st2chatops using 3.2dev on Centos8.