[Packager] Switch file watcher to Chokidar

[paulmillr]

Hey. You are currently using sane for file watching. How about using chokidar?

1. Community:
   • Sane was downloaded 98K times last month.
   • Chokidar was downloaded 58K times just yesterday. It is also used by socketstream and stuff like Gulp is switching to it right now.
2. Bugs:
   • Chokidar is obviously more tested in real-world applications - we have a solid track of 24m+ stable releases. We are not aware of any "node watch bugs" you mention on the web site — with correct implementation. Chokidar also has huge test coverage.
3. Speed:
   • With Chokidar, you may use our fastest available polling for OSX with FSEvents. Sane also uses FSevents, but through child_process. This is NOT fast and also error-prone. Chokidar uses FSevents with native calls.

I know that Sane is made by facebook dude, but how about trying more stable & robust solution?

[vkurchatkin]

#5 (comment)

[paulmillr]

@vkurchatkin

Additionally chokidar has so many problems but it's suffice to say it doesn't work on large codebases like ours.

They fixed a lot of issues in 1.0. I used to find it unreliable for months, but I suggest you give it another try unless you're absolutely sure.

He has the point. If chokidar has serious problems, why doesn't anyone report them? Are you absolutely sure that sane (which uses child_process) would handle large codebases without issues?

[amasad]

Hey. You are currently using sane for file watching. How about using chokidar?

We're primarily interested using watchman. Sane is just use for a fallback. Watchman is battle tested on facebook infrastructure, developer laptops and many open source users.

I'm still open to exploring that option but I believe that watchman is a much better technology.

Chokidar was downloaded 58K times just yesterday. It is also used by socketstream and stuff like Gulp is switching to it right now.

That's great. But by the same standards watchman is a much popular open source project than chokidar, it has 2.5k github stars while chokidar has 700 :)

Chokidar is obviously more tested in real-world applications - we have a solid track of 24m+ stable releases.

Again, watchman is also battle tested 36 releases :)

We are not aware of any "node watch bugs" you mention on the web site — with correct implementation. Chokidar also has huge test coverage.

Yes, because you use a native fsevents module.

[amasad]

He has the point. If we have serious problems, why doesn't anyone report them? Are you absolutely sure that sane (which uses child_process) would handle large codebases without issues?

I did report them in the past. It was incredibly slow to use on our codebase, I needed a ready event which I sent a pull request for that wasn't accepted. Even with the ready event it was no match to watchman's instant daemon model.

Also "child_process" is dishonest. We only exec to show an error message that watchman isn't installed.

[paulmillr]

Would be great to go full NPM without homebrew though — opened the issue facebook/watchman#91

Also "child_process" is dishonest.

Right. Sorry for that, the import confused me.

@amasad what about that?

  // We need to ask the client binary where to find it.
  // This will cause the service to start for us if it isn't
  // already running.
  var watchmanCommand = this.watchmanBinaryPath + ' get-sockname';
  childProcess.exec(watchmanCommand,

[es128]

@amasad Do you perceive that there would be value in adding a watchman option to chokidar for users willing to do the separate non-npm install?

If we were to move in that direction, is there potential for consolidating efforts toward a single library, or are there any other concerns/shortcomings with the recent versions of chokidar that would prevent you from considering this?

[amasad]

Would be great to go full NPM without homebrew though — opened the issue facebook/watchman#91

Maybe @wez can think of a way this could work. But the way watchman works -- and what makes it fast and reliable -- is that it's a daemon.

@amasad is that the exec check?

So this shells out to get the socket name and then just uses unix sockets to connect to watchman.

@amasad Do you perceive that there would be value in adding a watchman option to chokidar for users willing to do the separate non-npm install?

If we were to move in that direction, is there potential for consolidating efforts toward a single library, or are there any other concerns/shortcomings with the recent versions of chokidar that would prevent you from considering this?

Yes, this sounds great. If chokidar has all the backends/fallbacks that sane has then there is no reason for sane to exist.

[es128]

Yes, this sounds great. If chokidar has all the backends/fallbacks that sane has then there is no reason for sane to exist.

Well let me just finish pushing 1.0.0 out the door (should be next week), but after that I'd be happy to collaborate on giving this idea a try.

[amasad]

Here is the watching backends that would be great to support in this consolidated effort:

1. watchman -- used for high performance servers and large codebases
2. fsevents (what chokidar has) -- used for the common case where the user is on OS X and can install native module
3. node fs.watch (still better than polling) -- used for common linux and windows users
4. polling (currently done via the watch module on sane) -- used for environments where no native fs events exists, e.g. vagrant

Thoughts?

[es128]

By the way, one of the remaining pain points remaining on using chokidar with extremely large file trees isn't the actual watching (unless using polling) - it's the need to scan the entire tree at instantiation as a basis for normalizing events. Currently use readdirp for this, have been thinking about submitting a patch there to create a "fast mode" option that avoids getting a stat for each file.

[paulmillr]

@amasad sounds great

[es128]

Well yeah - chokidar already has great support for 2-4 on your list. The effort would be to add watchman, and preferably detect its presence to have it kick in by default when available - otherwise fall back to existing defaults.

[es128]

Also, regarding polling, see paulmillr/chokidar#242. Been mulling over how to solve this as well. Want to cover as many cases as possible where we can get it to "just work" without any environment-specific settings needed.

[wez]

I'd be happy to help you figure out how to consume watchman from chokidar. For large trees and multiple tools wanting to consume file notifications, using watchman as a consolidation point will reduce overall resource utilization and avoid node-process startup time overheads (particularly if you restart your node process).

[amasad]

@es128 cool! It seems like chokidar has really improved in the year that I hadn't looked. Let's make this happen.

cc @stefanpenner any concerns?