Skip to content

GetLocatedImports uses quadratic amount of memory in KnownTargets at startup #4317

Closed
@mpickering

Description

@mpickering

At startup GetLocatedImports is called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently.

In GetLocatedImports the known targets are consulted and the targetsMap is created by mapping the known targets. This map
is used for introducing sharing amongst filepaths. This operation copies a local copy of the target map which is local to the rule.

let targetsMap = HMap.mapWithKey const targets

So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be
10000^2 elements in total allocated in hashmaps. This used a lot of memory.

Solution: Return the normalising map in the result of the GetKnownTargets rule so it is shared across threads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: needs triagetype: bugSomething isn't right: doesn't work as intended, documentation is missing/outdated, etc..

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions