Dynamic linking problems?

[Vlix]

Not sure if this is an HLS issue, but building with stack itself works fine, so somewhere something is off.

The main problem seems to be somehow related to how we've built the tracing-newrelic package. I've added the LSP logs to show what's happening.

A few things to keep in mind:

• /home/user/project/scripts/files/libnewrelic.a is soft linked from /lib/libnewrelic.a
• If I move the libnewrelic.a from scripts/files, I get the following error:

  Unexpected usage error
  libnewrelic.so: cannot open shared object file: No such file or directory
  

• building with just stack build or stack ghci works fine
• I've sanitized a lot, because this is an internal company codebase.
• I've included only a few different errors I've gotten in the hls --debug snippet below, there are way more, but most only have the message Program Error:

I don't get why it doesn't work. If anyone could help me out, that'd be great!

---

A snippet from [haskell-language-server --debug]

File:     /home/user/project/package2/src/Module.hs
Hidden:   no
Range:    1:1-2:1
Source:   compiler
Severity: DsError
Message: 
  Data.HashMap.Base.(!): key not found
  CallStack (from HasCallStack):
  error, called at ./Data/HashMap/Base.hs:631:16 in
  unordered-containers-0.2.10.0-KujsThAgCnwGXBG7PJnkNO:Data.HashMap.Base
haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nr_close'
haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nrl_level_mask_ptr'
haskell-language-server: /home/user/.stack/snapshots/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/lib/x86_64-linux-ghc-8.8.3/tracing-newrelic-1.0.0.1-C9fxu7aaeN2Lo041riYaFq/libHStracing-newrelic-1.0.0.1-C9fxu7aaeN2Lo041riYaFq.a: unknown symbol `newrelic_ignore_transaction'
haskell-language-server: /home/user/project/.stack-work/install/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/lib/x86_64-linux-ghc-8.8.3/package1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA/libHSpackage1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA.a: unknown symbol `tracingzmnewreliczm1zi0zi0zi1zmC9fxu7aaeN2Lo041riYaFq_TracingziNewRelic_DatastoreSegment_con_info'
haskell-language-server: /home/user/project/.stack-work/install/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/lib/x86_64-linux-ghc-8.8.3/package1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA/libHSpackage1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA.a: unknown symbol `package1zm0zi2zi0zi0zm2gTTaf1DTx3EyQV3wpJlWA_DataziCompanyziEnvziDB_withSqlPersistT_closure'
haskell-language-server: /home/user/project/.stack-work/install/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/lib/x86_64-linux-ghc-8.8.3/package1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA/libHSpackage1-0.2.0.0-2gTTaf1DTx3EyQV3wpJlWA.a: unknown symbol `package1zm0zi2zi0zi0zm2gTTaf1DTx3EyQV3wpJlWA_UtilziDBziPersistent_insert_closure'
haskell-language-server: ^^ Could not load 'package1zm0zi2zi0zi0zm2gTTaf1DTx3EyQV3wpJlWA_UtilziError_logW_closure', dependency unresolved. See top entry above.

haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
File:     /home/user/project/package3/src/Module.hs
Hidden:   no
Range:    1:1-2:1
Source:   typecheck
Severity: DsError
Message:  Program error:
File:     /home/user/project/package4/src/Module.hs
Hidden:   no
Range:    1:1-2:1
Source:   typecheck
Severity: DsError
Message: 
  Program error:
  ByteCodeLink.lookupCE
  During interactive linking, GHCi couldn't find the following symbol:
  commonzm0zi2zi0zi0zm2gTTaf1DTx3EyQV3wpJlWA_UtilziError_logW_closure
  This may be due to you not asking GHCi to load extra object files,
  archives or DLLs needed by your current session. Restart GHCi, specifying
  the missing library using the -L/path/to/object/dir and -lmissinglibname
  flags, or simply by naming the relevant files on the GHCi command line.
  Alternatively, this link failure might indicate a bug in GHCi.
  If you suspect the latter, please send a bug report to:
  glasgow-haskell-bugs@haskell.org
haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
File:     /home/user/project/package5/src/Module.hs
Hidden:   no
Range:    1:1-2:1
Source:   typecheck
Severity: DsError
Message:  Program error:
haskell-language-server: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
File:     /home/user/project/package6/src/Module.hs
Hidden:   no
Range:    1:1-2:1
Source:   typecheck
Severity: DsError

LSP logs:

haskell-language-server version: 0.6.0.0 (GHC: 8.8.3) (PATH: /home/user/.local/bin/haskell-language-server-8.8.3) (GIT hash: 61c69fd387281c2222f75f13542eaa2e91cbc1b1)
Starting (haskell-language-server)LSP server...
  with arguments: LspArguments {argLSP = True, argsCwd = Nothing, argFiles = [], argsShakeProfiling = Nothing, argsTesting = False, argsExamplePlugin = False, argsDebugOn = False, argsLogFile = Nothing, argsThreads = 0, argsProjectGhcVersion = False}
  with plugins: [PluginId "brittany",PluginId "eval",PluginId "floskell",PluginId "fourmolu",PluginId "ghcide",PluginId "hlint",PluginId "importLens",PluginId "moduleName",PluginId "ormolu",PluginId "pragmas",PluginId "retrie",PluginId "stylish-haskell",PluginId "tactic"]
  in directory: /home/user/project
If you are seeing this in a terminal, you probably should have run ghcide WITHOUT the --lsp option!
 Started LSP server in 0.01s
2021-01-05 17:29:15.340933183 [ThreadId 21] - Registering ide configuration: IdeConfiguration {workspaceFolders = fromList [NormalizedUri 8153209061171765366 "file:///home/user/project"], clientSettings = hashed Nothing}
2021-01-05 17:29:15.342464754 [ThreadId 21] - Configuration changed: Object (fromList [("haskell",Object (fromList [("logFile",String ""),("updateBehavior",String "prompt"),("hlintOn",Bool True),("formatOnImportOn",Bool True),("indentationRules",Object (fromList [("enabled",Bool True)])),("liquidOn",Bool False),("languageServerVariant",String "haskell-language-server"),("serverExecutablePath",String ""),("diagnosticsOnChange",Bool True),("completionSnippetsOn",Bool True),("maxNumberOfProblems",Number 100.0),("formattingProvider",String "fourmolu"),("trace",Object (fromList [("server",String "messages")]))]))])
2021-01-05 17:29:28.396350009 [ThreadId 21] - Opened text document: file:///home/user/project/package1/src/PathToModule.hs
2021-01-05 17:29:28.397578522 [ThreadId 111] - getClientConfigAction:clientSettings:Just (Object (fromList [("haskell",Object (fromList [("logFile",String ""),("updateBehavior",String "prompt"),("hlintOn",Bool True),("formatOnImportOn",Bool True),("indentationRules",Object (fromList [("enabled",Bool True)])),("liquidOn",Bool False),("languageServerVariant",String "haskell-language-server"),("serverExecutablePath",String ""),("diagnosticsOnChange",Bool True),("completionSnippetsOn",Bool True),("maxNumberOfProblems",Number 100.0),("formattingProvider",String "fourmolu"),("trace",Object (fromList [("server",String "messages")]))]))]))
2021-01-05 17:29:28.397706219 [ThreadId 111] - hlint:getIdeas:file:NormalizedFilePath "/home/user/project/package1/src/PathToModule.hs"
2021-01-05 17:29:28.39781411 [ThreadId 105] - Consulting the cradle for "package1/src/PathToModule.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/home/user/project", cradleOptsProg = CradleAction: Stack}
2021-01-05 17:29:28.612669625 [ThreadId 121] - Plugin.makeCodeLens (ideLogger)
2021-01-05 17:29:28.8243354 [ThreadId 134] - GhcIde.hover entered (ideLogger)
2021-01-05 17:29:28.82454223 [ThreadId 134] - Hover request at position 54:20 in file: /home/user/project/package1/src/PathToModule.hs
> Configuring GHCi with the following packages: package1
> /home/user/project/.stack-work/install/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/pkgdb:/home/user/.stack/snapshots/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/pkgdb:/home/user/.stack/programs/x86_64-linux/ghc-8.8.3/lib/ghc-8.8.3/package.conf.d
2021-01-05 17:29:30.620428179 [ThreadId 105] - Using interface files cache dir: /home/user/.cache/ghcide/main-b4c0326a64fc0af4283707de7d9957e97233b004
2021-01-05 17:29:30.62083531 [ThreadId 105] - Making new HscEnv[main]
2021-01-05 17:29:30.794830543 [ThreadId 190] - getClientConfigAction:clientSettings:Just (Object (fromList [("haskell",Object (fromList [("logFile",String ""),("updateBehavior",String "prompt"),("hlintOn",Bool True),("formatOnImportOn",Bool True),("indentationRules",Object (fromList [("enabled",Bool True)])),("liquidOn",Bool False),("languageServerVariant",String "haskell-language-server"),("serverExecutablePath",String ""),("diagnosticsOnChange",Bool True),("completionSnippetsOn",Bool True),("maxNumberOfProblems",Number 100.0),("formattingProvider",String "fourmolu"),("trace",Object (fromList [("server",String "messages")]))]))]))
2021-01-05 17:29:30.795019385 [ThreadId 190] - hlint:getIdeas:file:NormalizedFilePath "/home/user/project/package1/src/PathToModule.hs"
2021-01-05 17:29:30.854360225 [ThreadId 274] - finish: CodeAction (took 0.06s)
2021-01-05 17:29:30.854760773 [ThreadId 273] - hlint:getIdeas:setExtensions:[Cpp,OverlappingInstances,UndecidableInstances,IncoherentInstances,UndecidableSuperClasses,MonomorphismRestriction,MonoPatBinds,MonoLocalBinds,RelaxedPolyRec,ExtendedDefaultRules,ForeignFunctionInterface,UnliftedFFITypes,InterruptibleFFI,CApiFFI,GHCForeignImportPrim,JavaScriptFFI,ParallelArrays,TemplateHaskell,TemplateHaskellQuotes,ImplicitParams,ImplicitPrelude,ScopedTypeVariables,AllowAmbiguousTypes,UnliftedNewtypes,BangPatterns,TypeFamilies,TypeFamilyDependencies,TypeInType,OverloadedStrings,OverloadedLists,NumDecimals,DisambiguateRecordFields,RecordWildCards,RecordPuns,ViewPatterns,GADTs,GADTSyntax,NPlusKPatterns,DoAndIfThenElse,BlockArguments,RebindableSyntax,ConstraintKinds,PolyKinds,DataKinds,InstanceSigs,ApplicativeDo,StandaloneDeriving,DeriveDataTypeable,AutoDeriveTypeable,DeriveFunctor,DeriveTraversable,DeriveFoldable,DeriveGeneric,DefaultSignatures,DeriveAnyClass,DeriveLift,DerivingStrategies,DerivingVia,TypeSynonymInstances,FlexibleContexts,FlexibleInstances,ConstrainedClassMethods,MultiParamTypeClasses,NullaryTypeClasses,FunctionalDependencies,UnicodeSyntax,ExistentialQuantification,MagicHash,EmptyDataDecls,KindSignatures,RoleAnnotations,ParallelListComp,MonadComprehensions,GeneralizedNewtypeDeriving,PostfixOperators,TupleSections,PatternGuards,LiberalTypeSynonyms,RankNTypes,ImpredicativeTypes,TypeOperators,ExplicitNamespaces,PackageImports,ExplicitForAll,AlternativeLayoutRuleTransitional,DatatypeContexts,NondecreasingIndentation,RelaxedLayout,TraditionalRecordSyntax,LambdaCase,MultiWayIf,BinaryLiterals,HexFloatLiterals,DuplicateRecordFields,OverloadedLabels,EmptyCase,PatternSynonyms,PartialTypeSignatures,NamedWildCards,TypeApplications,Strict,StrictData,MonadFailDesugaring,EmptyDataDeriving,NumericUnderscores,QuantifiedConstraints,ImportQualifiedPost,CUSKs,StandaloneKindSignatures,StarIsType]
2021-01-05 17:29:31.377399929 [ThreadId 273] - finish: CodeAction (took 0.00s)
2021-01-05 17:29:32.18367254 [ThreadId 1515] - finish: CodeAction:PackageExports (took 1.33s)
2021-01-05 17:29:32.1836766 [ThreadId 1516] - finish: CodeAction:PackageExports (took 0.81s)
2021-01-05 17:29:32.184178174 [ThreadId 1517] - finish: importLens (took 0.00s)
2021-01-05 17:29:33.313416899 [ThreadId 1787] - finish: codeLens (took 2.52s)
2021-01-05 17:29:33.313472475 [ThreadId 1790] - finish: retrie (took 1.13s)
2021-01-05 17:29:33.314263719 [ThreadId 1795] - finish:  (took 0.00s)
2021-01-05 17:29:33.314910113 [ThreadId 1796] - finish: ModuleName.ghcSession (took 0.00s)
2021-01-05 17:29:33.315274003 [ThreadId 1798] - finish: ModuleName.GetParsedModule (took 0.00s)
2021-01-05 17:29:33.316240671 [ThreadId 1802] - finish: tactic (took 0.00s)
2021-01-05 17:29:33.316523823 [ThreadId 1806] - finish: tactic (took 0.00s)
2021-01-05 17:29:33.317761669 [ThreadId 1807] - finish: tactic (took 0.00s)
2021-01-05 17:29:33.719555595 [ThreadId 1926] - finish: CodeAction (took 0.00s)
2021-01-05 17:29:33.719629204 [ThreadId 1808] - finish: CodeAction:PackageExports (took 0.00s)
2021-01-05 17:29:33.720143957 [ThreadId 1927] - finish: importLens (took 0.00s)
2021-01-05 17:29:33.720357099 [ThreadId 1928] - finish: retrie (took 0.00s)
2021-01-05 17:29:33.720563996 [ThreadId 1929] - finish: tactic (took 0.00s)
2021-01-05 17:29:33.72069339 [ThreadId 1929] - finish: tactic (took 0.00s)
2021-01-05 17:29:33.720784285 [ThreadId 1929] - finish: tactic (took 0.00s)
haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nr_close'
haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nrl_level_mask_ptr'
haskell-language-server-8.8.3: /home/user/.stack/snapshots/x86_64-linux/500986717783a68e606e0b2a92ead5d49774d1810b2804e9f52dbdead82b7e1f/8.8.3/lib/x86_64-linux-ghc-8.8.3/tracing-newrelic-1.0.0.1-C9fxu7aaeN2Lo041riYaFq/libHStracing-newrelic-1.0.0.1-C9fxu7aaeN2Lo041riYaFq.a: unknown symbol `newrelic_ignore_transaction'
haskell-language-server-8.8.3: ^^ Could not load 'tracingzmnewreliczm1zi0zi0zi1zmC9fxu7aaeN2Lo041riYaFq_TracingziNewRelic_DatastoreSegment_con_info', dependency unresolved. See top entry above.

haskell-language-server-8.8.3: ^^ Could not load 'tracingzmnewreliczm1zi0zi0zi1zmC9fxu7aaeN2Lo041riYaFq_TracingziNewRelic_DatastoreSegment_static_info', dependency unresolved. See top entry above.

[adamse]

Is /home/user/.local/bin/haskell-language-server-8.8.3 a mostly static build? It looks like the GHC RTS linker is failing to load libnewrelic.a.

haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `stat'
haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nr_close'
haskell-language-server-8.8.3: /home/user/project/scripts/files/libnewrelic.a: unknown symbol `nrl_level_mask_ptr'

The stat symbol has a workaround in the latest GHC 8.10 release (https://gitlab.haskell.org/ghc/ghc/-/issues/7072), the other two I'm less sure about.

Maybe you can work around it by manually adding libc_nonshared.a and whatever library nr_close and nrl_level_mask_ptr comes from to the libraries you depend on.

You could also try your own dynamic build of haskell-language-server-8.8.3.

[Vlix]

Oh man. I upgraded to HLS 0.8.0 (using ghcup), and now haskell-language-server-8.8.3 --debug is segfaulting again. 😞

@adamse nr_close and nrl_level_mask_ptr should come from libnewrelic.a, so I don't know how i'd go about fixing that.

And where would I build the dynamic build of HLS-8.8.3?

[jneira]

you can build it from source, maybe that alone could work

[Vlix]

Ok, so... that's managed to fix this, and I'm not sure what the cause was but building HLS-0.8.0 from source seems to have worked.
Using ghcup's HLS-0.8.0 kept segfaulting on TH splices, but after building from source, it now seems to just work again. without linking errors.

[Vlix]

Woops, seems like I screwed up...

I think I've tried the source-built HLS in a different branch. The branch that I had trouble with still has the same unknown symbol errors.

[Vlix]

I suspected it had something to do with the extra-deps being different (the branch that doesn't work has a different commit for tracing-newrelic in the extra-deps) and stack repl {package1} shows this now (it didn't previously)

In file included from <command-line>:9:0: error: 

/tmp/ghc28361_0/ghc_2.h:325:0: error:
     warning: "VERSION_tracing_newrelic" redefined
     #define VERSION_tracing_newrelic "1.0.0.1"
     
    |
325 | #define VERSION_tracing_newrelic "1.0.0.1"
    | ^

In file included from <command-line>:9:0: error: 

/home/user/project/.stack-work/ghci/898d05a8/cabal_macros.h:610:0: error:
     note: this is the location of the previous definition
     #define VERSION_tracing_newrelic "1.0.0.0"
     
    |
610 | #define VERSION_tracing_newrelic "1.0.0.0"
    | ^

(but the above times five (x 5) with the ghc_%d.h with %d `elem` [2,4,6,8,10])

[Vlix]

Bit more fiddlin' around. Doesn't seem to be the version difference. It's something in our code (because it's really just this one branch, when given the same tracing-newrelic commit hash in the extra-deps.)

[mpickering]

Have you tried building a dynamically linked hls?

[Vlix]

Have you tried building a dynamically linked hls?

@mpickering If you could tell me how, I'd appreciate it. Would that work with just stack build --ghc-options="{insert dynamic option}"?

[Naveenaidu]

@Vlix I'm not sure if this is of any help, I had the same kind of problem and I had to do the following to get it working:

1. Make changes to the haskell-language-server.cabal file present in the source with the following(Attaching the diff below):

diff --git a/haskell-language-server.cabal b/haskell-language-server.cabal
index dcc4a6c..1e6a2c9 100644
--- a/haskell-language-server.cabal
+++ b/haskell-language-server.cabal
@@ -70,7 +70,7 @@ library
     , shake
     , unordered-containers
 
-  ghc-options:      -Wall -Wredundant-constraints -Wno-name-shadowing
+  ghc-options:      -Wall -Wredundant-constraints -Wno-name-shadowing -dynamic
 
   if flag(pedantic)
     ghc-options: -Werror
@@ -273,7 +273,7 @@ executable haskell-language-server
   other-modules:    Plugins
 
   ghc-options:
-    -threaded -Wall -Wno-name-shadowing -Wredundant-constraints
+    -threaded -Wall -Wno-name-shadowing -Wredundant-constraints -dynamic
     -- allow user RTS overrides
     -rtsopts
     -- disable idle GC
@@ -320,7 +320,7 @@ executable haskell-language-server-wrapper
   other-modules:    Paths_haskell_language_server
   autogen-modules:  Paths_haskell_language_server
   ghc-options:
-    -threaded -Wall -Wno-name-shadowing -Wredundant-constraints
+    -threaded -Wall -Wno-name-shadowing -Wredundant-constraints -dynamic
     -- allow user RTS overrides
     -rtsopts
     -- disable idle GC

2. Build the hls again. Since I used cabal to install HLS from source I ran the following:

./cabal-hls-install hls-8.10.2 

hls-8.10.2 because we use that ghc version. I guess the stack equivalent to the above command would be stack --stack-yaml stack-8.10.2.yaml build

Hopefully this helps 🤞

[Vlix]

Ok, I don't want to celebrate prematurely again, but building it with the -dynamic flags does seem to have fixed the problem. 😄
(Just changed the haskell-language-server.cabal and ran stack ./install.hs hls-8.8.3)