Skip to content
This repository was archived by the owner on Oct 7, 2020. It is now read-only.
This repository was archived by the owner on Oct 7, 2020. It is now read-only.

Overhaul test-suites #1585

Open
Open
@fendor

Description

@fendor

A single test-run takes a huge amount of time and, by now, we have a huge array of VM tests.
Currently, we have the resulting matrix of test-runs:

Tool\GHC 8.4.2 8.4.3 8.4.4 8.6.1 8.6.2 8.6.3 8.6.4 8.6.5 8.8.1
Cabal ✔️ ✔️ ✔️
Stack ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️

Resulting in 12 VMs to just build in every configuration, making it 36 VMs for every supported operating system: Linux | MacOs | Windows.

Tests take roughly 20 minutes or even more (on my machine 26 min) to run.

To make the matter worse, we should have tests where only either cabal or stack is installed and both.

What I want to discuss in this post is an overhaul of the test system with the following points:

  • Functional tests import suggestions are incredibly slow
  • Prefer direct cradle over stack cradle
    • On my system a call to stack repl takes roughly 1.5s,
      • I estimate we have 140 calls to stack repl, replacing this with a direct cradle will reduce run-time by two minutes at least
  • Add tests for loading project context with stack and cabal, similar to cabal-helper.
    • Since hie-bios abstracts the build-tools, we dont have to execute all tests for each build-tool as long as we test that the build-tools initialize the context correctly and provide compilation arguments.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions