Iterators and for-of loops

[cometkim]

This proposal introduces new syntax and semantics for improving for loops.

Motivation

Currently, ReScript has very limited for loops support.

In particular, the lack of support for iterators makes interacting with data structures on the JS side difficult. ReScript libraries provide function-style utilities such as forEach and reduce to make it possible. However, it potentially has several problems.

1. Poor performance: in many cases forEach and reduce are slower than the native for loop. This is a problem in hot paths where micro-optimization is required.
2. Memory usage: The ReScript core has an explicitly typed Core__Iterator, but this is practically useless. To use it in ReScript user has to convert it to an array via toArray, which adds unnecessary memory footprints.

These can be solved with hard-written for/while statements, but it is not a recommended solution.

@chenglou mentioned that we need some ergonomic improvements on it.
https://twitter.com/_chenglou/status/1653739852347879424

Presumption

Iterators are not compatible with the semantics of records.

ReScript itself has no class instances and it's always coming from the JS side.

Converting to iterators

First, add opaque types Js.iterator<'t> and Js.asyncIterator<'t> to core. This means the object available for the for-of loop and its entry type is 't.

Worth noting that Iterators are not always Array-like, such as Map

let map = new Map();
map.set(1, 1);
map.set(2, 2);

for (const [key, val] of map) {
  console.log(key, val);
}
// 1 1
// 2 2

ReScript types are:

// Map.res

type t<'key, 'val>

type iterator<'key, 'val> = Js.iterator<('key, 'val)>

There are two paths to getting a map's iterator: the map instance itself and Map.prototype.entries. ReScript needs to be able to express both of these as bindings.

@iterator
external iter: t<'key, 'val> => iterator<'key, 'val> = "%identity"

@iterator @get
external entries: t<'key, 'val> => iterator<'key, 'val> = "entries"

the @iterator tag if it helps the compiler implementation, to validate if the return value is Js.iterator.

Using iterators in for-of loops

Where possible, avoid adding new keywords here. Especially of as it is already a popular name in existing ReScript codebases.

Maybe add Js.iterator support to existing for ... in loops.

for entry in map->iter {
  ...
}

Also add destructuring

for (key, val) in map->iter {
  ...
}

Syntax looks similar to popular languages like Python or Kotlin, except we call iter function here.

It might be confused with js for-in syntax. We need to specify/teach that this compiles with for-of

Almost same for async iterators, but with await syntax. For example Deno HTTP server:

let server = Deno.listenWithOptions({ port: 8080 })
for await conn in server->Server.asyncIter {
  let httpConn = conn->Deno.serveHttp
  for await event in conn->HttpConn.asyncIter {
    // handle request event...
  }
}

Further improvements

Data-first syntax

The traditional for loop syntax is not data-first. We might also consider making additional syntax changes to better interact with ReScript's type inference.

example:

for map->iter to (key, val) {
  ...
}

So entries can be properly type-checked at input time

Break, continue

Since we don't share semantics with OCaml anymore, I think there's no reason not to introduce it.

One possible problem is if the user tries to break in a pattern match statement. break can be nested in compiled switch-case statements, destroying their semantics.

[cristianoc]

If one does iter and entries with %raw, how much of this can already be done today?

[chenglou]

for-loop is definitely a sore spot in the language right now. I think if we're thinking of beefing it up, we should start with break and continue first before thinking of iterators. Or tbh maybe comprehension a-la python.

I'm not too sure about iterators. Those interfaces are kinda not too ergonomics in almost all the languages I've seen. Jai/C uses a macro, which is nice and often enough. If we introduce some mechanism for it, then at least the syntax shouldn't be a departure from JS. Otherwise it's in this weird sour spot of being not that much better than FP helpers, not enough control flow (no break/continue), not that fast, etc.

[CarlOlson]

If one does iter and entries with %raw, how much of this can already be done today?

I've had bindings for a long time. You can already use iterators in a performant way. I don't think new syntax is necessary:

module Map = {
  type t<'k, 'v>

  @send external values: t<'k, 'v> => Iterator.t<'v> = "values"
  @send external keys: t<'k, 'v> => Iterator.t<'k> = "keys"
  @send external entries: t<'k, 'v> => Iterator.t<('k, 'v)> = "entries"
}

Then I wrote helpers (map, filter, etc) for Iterator and AsyncIterator in JS with ReScript bindings (this was before the JS parser was updated). AsyncIterator allowed me to stream Redis results without blowing up the heap.

for-loop is definitely a sore spot in the language right now. I think if we're thinking of beefing it up, we should start with break and continue first before thinking of iterators.

I don't really see that. I work in a large codebase that has gone through many style evolutions. For-loops and while-loops are only used in our standard library and only a handful of times. Not by making it a rule, just naturally, readable ReScript doesn't use those constructs.

break and continue aren't necessary by using while. The places we use while are actually hardly used or (the ones I wrote) poorly designed 😅

Think about what you would rather see? A complex loop or a well-structured recursive function. Adding "goto" constructs to the language is a step back.

Or tbh maybe comprehension a-la python.

This seems like feature envy and would fragment coding styles. I don't see an advantage in real-world code.

---

An alternative to all this would be to solidify your PPX alternative. I'm sure people would come up with great ideas that could then be added to ReScript after being proven useful.

[chenglou]

The major point of controlling the syntax is precisely to rectify what should and should not be readable. Starting from what's currently readable and using that to decide further semantics has historically not worked out well for us, considering lots of these arguments are motivated by achieving functional nicety over practicality, rather than the other way around (the alter of which is what we're mainly defined by, even after all this time).

break and continue are very different than goto, and we're well aware of these structured programming arguments. Complex loop vs well-structured recursion aren't the right comparison. Plus recursion debugging is hard when we optimize them into tail calls. This argument has been hashed out a lot previously already elsewhere, e.g. https://twitter.com/deech/status/1567891572871643136

Anyway, I'd like to come back on topic.

• I'd prefer us not introduce new concepts for iterators, unless there's big benefits, and so far it seems to me that the benefits would be in some potential break and continue, which the function helpers don't have (and let's not wrap those with some option or whatever to indicate stopping). Plus familiarity with JS (for-of without break/continue would be more of a negative surprise tbh).
• But I'm also not even sure we should add break and continue, due to various semantic reasons like the aforementioned pattern matching.
• But if we do consider custom iterations (whether it's a macro a-la C or some construct a-la Swift/Rust), we should revamp the for-loop syntax instead of having iterators tackled onto the current (not very good) for-loop syntax. For example, we could remove for-in-to.

[cometkim]

For-loops and while-loops are only used in our standard library and only a handful of times. Not by making it a rule, just naturally, readable ReScript doesn't use those constructs.

In addition to cases where loops contain break/continue, there are also cases where await is needed in the middle of the loop.

So for example in JS:

for (const item of items) {
  try {
    await doAsyncTaskInSequence(item)
  } catch (e) {
    console.warn(e);
    // break
  }
}

It would be translated as follows:

items->Iterable.reduce(Promise.resolve(), (promise, item) => {
   promise
   ->Promise.then(_ => {
     doAsyncTaskInSequence(item)
     ->Promise.catch(e => {
        Console.warn(e)
        // no direct way to break
     })
   })
})

I don't understand how this kind of code flow isn't common. Of course, it can be avoided somehow, but the code architecture for that should be considered from the very beginning.

Maybe users need to introduce an external runtime like Fluture or Rx. Which is not "natural", And it is a huge setback to the program flow and debugging experience.

[CarlOlson]

It would be translated as follows:

items->Iterable.reduce(Promise.resolve(), (promise, item) => {
   promise
   ->Promise.then(_ => {
     doAsyncTaskInSequence(item)
     ->Promise.catch(e => {
        Console.warn(e)
        // no direct way to break
     })
   })
})

Just catch it at a higher level... no need for a break:

switch await items->Iterable.forEachAsync(doAsyncTask) {
| () => ...
| exception exn => ...
}

[ellbur]

Maybe as a first iteration (no pun intended), a simple syntactic sugar for an anonymous recursive function would make sense?

That way the semantics would be crystal clear, async would just work, it could build complex result values, and it could be extended to more complex traversable structures.

Various possible syntaxes might work, all getting at the same basic idea:

for items { (item, next) =>
  Console.log(item)
  next()
}

for tree { (node, next) =>
  Console.log(node)
  next(node.left)
  next(node.right)
}

for stream { async (item, next) =>
  Console.log(item)
  for item.subitems { async (subitem, next2) =>
    Console.log(subitem)
    await next2()
  }
  await next()
}

[CarlOlson]

Maybe as a first iteration (no pun intended), a simple syntactic sugar for an anonymous recursive function would make sense?

I don't think the implementation is an issue. The issue is more about getting consensus. ReScript is pretty conservative with development, a work-in-progress wouldn't be desirable for teams building products.

If you want to use iterators... The "iterator helper" changes to JS will help use iterators with ReScript's pipe syntax without overhead (partly implemented in Node 22). Or they can be implemented now with %raw:

let flatMap: (t<'a>, @uncurry ('a => t<'b>)) => t<'b> = %raw("
  function (t, fn) {
    return {
      *[Symbol.iterator]() {
        for (const value of t) {
          yield* fn(value);
        }
      },
    };
  }
")

The only thing we can't do is a break or continue without resorting to exceptions or checking the return value of the fn function above (could be non-trival if using options due to nesting). But implementing these yourself could have performance/debugging issues depending on usage.

However, I would like to see syntax for iterators, generator functions, and yield eventually. :)