[native/js/wasm] Platform independent File I/O

[dcharkes]

Current situation

Dart's file IO capabilities are fragmented across different platforms and mechanisms. The dart:io library provides comprehensive file handling for native platforms but all methods throw in web dart2js and dart2wasm. IOOverrides enable overriding a subset of the APIs, but dart:io contains many more APIs that might not be implementable in the wasm and js backends.

void main(List<String> args) {
  IOOverrides.runWithIOOverrides(() {
    // ...
  }, MemFSIOOverrides());
}

MemFSIOOverrides

import 'dart:io';
import 'dart:js_interop';
import 'dart:convert' as convert;
import 'dart:js_interop_unsafe';
import 'dart:typed_data';

// adapted functions from https://emscripten.org/docs/api_reference/Filesystem-API.html#id2
extension type MemFS(JSObject _) implements JSObject {
  external JSArray<JSString> readdir(String path);
  external JSUint8Array readFile(String path, [JSObject? opts]);
  external void writeFile(String path, String data);
  external void unlink(String path);
  external void mkdir(String path);
  external void rmdir(String path);
  external void rename(String oldpath, String newpath);
  external String cwd();
  external void chdir(String path);
  external JSObject analyzePath(String path, bool dontResolveLastLink);
}

@JS('FS')
external MemFS get memfs;

class MemFSDirectory implements Directory {
  @override
  String path;

  MemFSDirectory(this.path);

  @override
  void createSync({bool recursive = false}) {
    memfs.mkdir(path);
  }

  @override
  dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}

class MemFSFile implements File {
  @override
  String path;

  MemFSFile(this.path);

  @override
  MemFSFile get absolute => MemFSFile(path);

  @override
  void createSync({bool recursive = false, bool exclusive = false}) {
    memfs.writeFile(path, '');
  }

  @override
  void deleteSync({bool recursive = false}) {
    memfs.unlink(path);
  }

  @override
  bool existsSync() {
    return memfs
        .analyzePath(path, false)
        .getProperty<JSBoolean>('exists'.toJS)
        .toDart;
  }

  @override
  void writeAsStringSync(String contents,
      {FileMode mode = FileMode.write,
      convert.Encoding encoding = convert.utf8,
      bool flush = false}) {
    memfs.writeFile(path, contents);
  }

  @override
  Uint8List readAsBytesSync() {
    return memfs.readFile(path).toDart;
  }

  @override
  String readAsStringSync({convert.Encoding encoding = convert.utf8}) {
    return encoding.decode(readAsBytesSync());
  }

  @override
  dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}

class MemFSIOOverrides extends IOOverrides {
  @override
  MemFSDirectory createDirectory(String path) {
    return MemFSDirectory(path);
  }

  @override
  MemFSFile createFile(String path) {
    return MemFSFile(path);
  }

  @override
  bool fsWatchIsSupported() {
    return false;
  }

  @override
  void setCurrentDirectory(String path) {
    memfs.chdir(path);
  }

  @override
  MemFSDirectory getCurrentDirectory() {
    return MemFSDirectory(memfs.cwd());
  }

  @override
  MemFSDirectory getSystemTempDirectory() {
    return MemFSDirectory("/tmp");
  }
}

Thanks @TheComputerM! 🚀

If users want to write code that uses file IO that works across multiple backends, this is what they currently can use.

Problems

• Not all dart:io APIs might be implementable on all backends. (locks? sockets? stdio? http requests?)
• Setting up IOOverrides is not the cleanest of solutions, it requires modifying the main function.
• (We have an exploration going on of trying to unbundle dart:io into package:io cc @brianquinlan)

Proposed solution

Introduce a File API in a package, potentially package:file, that works seamlessly across all Dart platforms. The new API would expose a subset of file operations supported by all targets, enabling developers to write platform-agnostic code for essential file interactions.

We should then be opinionated and tell users to use that package instead of dart:io directly.

This is how it's done with http, we have package:http which works on all platforms and it uses conditional imports to use dart:io on VM and dart:html on web.

I believe package:file is currently missing support for using the file system on the web.

• https://github.com/google/file.dart/issues/135

Using the memory file system works for producing (temporary) files in Dart, but does not work for:

• mounting files from the context
• invoking file APIs from C code compiled to WASM

Some open questions:

• Is package:file the right place to provide the platform abstraction?
• What part of mounting files can be done from the embedder or from Dart code? (I believe dart2js already has a list of callbacks somewhere similar to how the DartVM has a list of callbacks for IO?) (A CallbackFileSytem would allow wiring up from inside Dart, but it's probably preferable to do this at the embedder level.)

Soliciting input from

• package:file @jonahwilliams @jakemac53
• dart2js @sigmundch @kevmoo @natebiggs
• dart2wasm @mkustermann @osa1
• dart:io boss @brianquinlan
• Language team? @lrhn

Any feedback is welcome. Should we take a different direction?

Thanks @mkustermann for suggesting this approach. 🙏 And thanks @TheComputerM for bringing this issue up! 🚀

Let's enable our users to write code with file I/O that works everywhere.

[jakemac53]

• Is package:file the right place to provide the platform abstraction?

It probably does make sense yes, I don't see why not. It already exists and has a good name, plus much of it is already written. Note that I don't actually work on the package though (I think it has no official owner currently). We would probably want to more officially support it and figure out an owner.

Using this package also makes testing easier/faster because you can use an in memory file system.

[lrhn]

The dart:io library provides comprehensive file handling for native platforms but all methods throw in web dart2js and dart2wasm

It's a hack that you can import dart:io on those platforms at all, it's not intended to be available or supported.
(And dart.library.io is not set to true in the compilation environment.)

So a new API is a better idea for supporting other platforms.

The new API would expose a subset of file operations supported by all targets

Maybe just "operations".

Is "file" even the best abstraction? It's not Unix, not everything has to be a file.
It's something that can contain ... other things. But so is a database.

A file system is contains

• "files" (sequences of bytes)
• structure (hierarchical, traditionally, providing a way to identify files.)
• metadate. File kinds, bit flags, timestamps.
• possibly mutable (but not if mounted read-only).

If we are going to abstract over "file system" to platforms that are not POSIX, not operating systems, and has no real file system, it might be worth upping the abstraction level.

And figure out which subset of operations is really needed, without trying to support too much.

(What does "mounting files" mean?)

[dcharkes]

Is "file" even the best abstraction? It's not Unix, not everything has to be a file.

In the context where we use it, native code compiled to WASM with emscripten, it is a Posix-like file API. https://emscripten.org/docs/api_reference/Filesystem-API.html

Trying to use a different abstraction than a file system will not work if the native code is also trying to use the file system. We need to write a file and then pass in a file path to the native code.

It's a hack that you can import dart:io on those platforms at all, it's not intended to be available or supported.

Precisely. That's why should come up with something for users that is intended to be used.

[mkustermann]

it might be worth upping the abstraction level.

I think there's actually a need to go higher in abstraction and lower: Example: With our existing dart:io APIs it's not possible for a programmer know why a file cannot be opened for writing (e.g. the fact that it doesn't have permission - all errors are just FileSystemExceptions).

What would be nice is to have

• high-level abstractions that would abstract over platform differences, packages like package:file / package:http / ... that work everywhere
• low-level access to platform-specific features, package:posix, package:winapi, ...
  => can be implemented via C FFI (missing features can be implemented - shared memory multithreading will be one)
• make the high-level package use conditional imports to use the low-level ones
• start slimming down & eventually removing dart:io

[jakemac53]

Note that package:file does implement the existing dart:io types (like File, Directory, etc).

So, it currently has the exact same layer of abstraction as dart:io for the most part. That might make it not a good fit, because it means it requires a (transitive) dart:io dependency to use, and might have abstractions that don't make sense on all platforms.

[sigmundch]

Overall I like this direction and I'd love to eventually remove dart:io from the web backends :)

[kevmoo]

From a browser perspective (JS/Wasm) we're trying to AVOID supporting concrete types in dart: libraries. We're moving everything to https://pub.dev/packages/web and trying to deprecate dart:html (and friends).

Having interfaces that can be implemented is a GREAT idea. I think pkg:http does a good job here.

But I worry about making promises about implementations.

A dart:file implementation for https://developer.mozilla.org/en-US/docs/Web/API/File might be different than an implementation that works for https://github.com/WebAssembly/wasi-filesystem

The important thing is having the API surface that can be shared across use cases but backed by any implementation.

(I originally read this as wanting dart:file. Glad I misread. Picking a package to invest in here sounds GREAT. We'd have to chat about the best ways to expose web bits, but I could imagine a ~straightforward implementation backed by pkg:web, etc)