Error Unquoting deftype across namespaces in macros

[ikappaki]

Hi,

When trying to unquote a deftype defined in namespace B from namespace A in a macro, the following error occurs:

NameError: name 'TypeOther' is not defined

To reproduce

1. Create an other.lpy file defining a new type
   other.lpy

(ns other)

(deftype TypeOther []
  (__repr__ [this] "a"))

(TypeOther.)

2. Create an issue.lpy file that uses the type in a macro

(ns issue
  (:require other)
  (:import other))

(println :type-other (other/TypeOther))

(defmacro issue []
  `(identity ~(other/TypeOther)))

(println :issue (issue))

4. on the REPL, import the issue namespace, the above error is thrown

basilisp.user=> (require 'issue)
:type-other a
Traceback (most recent call last):
 ...
  File "C:\src\basilisp\src\issue.lpy", line 10, in <module>
    (println :issue (issue))
                    ^^^^^^^
NameError: name 'TypeOther' is not defined

The same works in Clojure
other.clj

(ns other)

(deftype TypeOther []
  Object
  (toString [this] "a"))

issue.clj

(ns issue
  (:require other)
  (:import [other TypeOther]))

(defmacro issue []
  `(identity ~(TypeOther.)))

(println :issue (issue))

user=> (require 'issue)
:issue #object[other.TypeOther 0x3b64ccce a]
nil

Note that to reference the deftype in Clojure, the other namespace must be both :required and the type explicitly :imported.

Encountered while trying to port the hiccup lib to Basilisp for fun in case you are wondering where the latest corner cases are coming from 😅. It goes well so far (praise to your hard work maintaining compatibility with Clojure 🙌), I can compile the lib fine, I'm currently running through the tests. Thanks for the latest fixes.

[chrisrink10]

Interesting. NameErrors come from the Python VM. So probably the Basilisp compiler is ok with this snippet but emits Python code which makes a direct reference to the name TypeOther rather than a namespaced reference.

I'll have to dig into this more when I have a minute.

[chrisrink10]

I was finally able to figure out what's going wrong here -- just need to figure out a fix for it!

The compiler tries to generate a reference to the type using __qualname__ only, but that is only qualified with respect to the module it's defined in, not the sys.path or what modules are imported within a namespace.

[chrisrink10]

Basilisp does a lot of special casing for generating code for "constants" (probably a bad name for what they are since they aren't necessarily constant). We have to be able to write a value to bytecode or otherwise it cannot be cached!

I have a POC PR which just uses pickle as the default for constants if they do not have otherwise have a default handler defined. That seems to fix the issue you raised and will probably handle most other cases pretty cleanly (with a few exceptions).

[ikappaki]

Basilisp does a lot of special casing for generating code for "constants" (probably a bad name for what they are since they aren't necessarily constant). We have to be able to write a value to bytecode or otherwise it cannot be cached!

I have a POC PR which just uses pickle as the default for constants if they do not have otherwise have a default handler defined. That seems to fix the issue you raised and will probably handle most other cases pretty cleanly (with a few exceptions).

Thanks, I've tested this and it works perfectly, solving the hiccup test suite break 🙌 It also eliminates the need to both require and import where the deftype is defined in. Now, (require '[other :refer [TypeOther]]) alone does the trick, which I think is how it should have worked in Clojure as well. So, this is even an improvement over Clojure 😅

I'll leave the Bandit pickle security report to you 😅 It looks like a valid case to me, as the serialisation/deserialisation happens within the same compilation unit and is not generated by an external process.

That said, do you think that a pickled defintion generated with one Python version might not be compatible if read by another? or is it the cases that these bytecodes are always read by the same Python version?

[chrisrink10]

I'll leave the Bandit pickle security report to you 😅 It looks like a valid case to me, as the serialisation/deserialisation happens within the same compilation unit and is not generated by an external process.

I'm going to suppress this warning. I think it's probably ok, but I'm sure there's some possible security angle for a dedicated attacker.

That said, do you think that a pickled defintion generated with one Python version might not be compatible if read by another? or is it the cases that these bytecodes are always read by the same Python version?

Python bytecode is not guaranteed to be stable across versions and it is cached by version, so this should be fine.