[Proposal/RFC] Split Parse Server Extensions from Core Repo

[asciimike]

Recently there's been a lot of activity in creating extensions to Parse Server's capabilities in the form of third party connectors to different services: Files (#545, #708, #716), Push (#711), Analytics (#719), and Mail (#627), and the Database (starting with #698) have all seen contributions to add new services or interest in modular components. This is awesome, as it shows some serious flexibility as well as community commitment--keep it up!

That said, as we start adding more and more extensions to the core feature, we're starting to encounter a few problems:

1. Testability and key management: Thanks to @flovilmart, a simple framework for testing multiple adapters exists (9287afc)! Unfortunately, testing third party services on the shared Parse infrastructure is increasingly difficult due to key management--unless the Parse Server Travis server maintains keys for all the services (which is untenable), CI tests on third party modules can't easily exist. Heavy mocking of each service could also solve this problem, though this is a challenge and a barrier to entry.
2. Imports: As an example, we'll soon be importing AWS, Azure, and GCP node modules to allow for cloud hosted file storage--which seems like a bit of overkill as it's unlikely that a developer will want to use all three (let alone any one of the three). If every feature (push, analytics, mail, etc.) has the top three modules involved, we'll soon be having quite the party downloading half of NPM every time we want to update our server ;)
3. Onboarding/ease of creation: As more and more extensions get added to the core codebase, we risk making it harder to maintain the existing code, as well as make it harder to add extensions, since the knowledge required to work in the codebase becomes higher. Modularizing code by creating similar, standalone chunks means developers interested in contributing to an area can more quickly and easily dive in to help.

With those problems in mind, I have seen/heard of (at least) three ways of approaching them:

1. Split functionality up into use-case specific extensions: for instance, parse-server-extensions-files, parse-server-extensions-push, parse-server-extensions-mail, etc.
2. Split functionality up into provider specific extensions: for instance, parse-server-provider-aws, parse-server-provider-azure, parse-server-provider-gcs, etc.
3. Make everything independent: parse-server-extension-gcsadapter, parse-server-extension-sns, and parse-server-extension-foo all live independently. We could provide a set of base templates to help get started which feature basic jasmine tests and a parse-server testing rig (say a parse-server-extension-template repo.

1 has the advantage of providing a logical grouping of services which promotes testability and makes it logically easier for the related code to stay in lock step. It also lowers the barrier to entry for developers looking to add new features, since there are a number of well tested examples that can be easily added upon at once, and there's no need to seek approval from a whole number of different people in order to add a cross provider feature (say like what happened with createBucket() in FilesAdapter).

In this model, code could be imported to parse-server like so:

var S3Adapter = require('parse-server-extensions-files').S3Adapter(...);
var AzureBlobAdapter = require('parse-server-extensions-files').AzureBlobAdapter(...);
var GCSAdapter = require('parse-server-extensions-files').GCSAdapter(...);

2 has the advantage of solving key management and imports far better, since it is ideally a single set of keys per provider, but I think it makes it harder for individual devs to contribute to a new extension (or a major feature on an existing extension) given that one would potentially have to talk to many more people and consensus gathering would be harder.

In this model, code could be imported to parse-server like so:

var S3Adapter = require('parse-server-provider-aws').S3Adapter(...);
var AzureBlobAdapter = require('parse-server-provider-azure').AzureBlobAdapter(...);
var GCSAdapter = require('parse-server-provider-gcp').GCSAdapter(...);

3 is the wild west approach, and allows the ultimate flexibility with the ultimate potential for confusion and lack of discoverability. Obviously anyone can already do this, but the barrier to entry is much higher. A template repo would help significantly, but might still result in similar feature sets or providers behaving differently from feature to feature.

In this model, code could be imported to parse-server pretty much however...

var S3Adapter = require('sams-s3-adapter').S3Adapter(...);
var AzureBlobAdapter = require('alexas-azure-blob-adapter').AzureBlobAdapter(...);
var GCSAdapter = require('garys-gcs-adapter').GCSAdapter(...);

What are the other options I'm missing that are worth discussing?

Orthogonal to this discussion would be (aka, doesn't have to be discussed now), once things are pulled out, would there be a standard set of services packaged with parse-server core: say Mongo, S3, GCM, etc. (original services)?

I know the core maintainers have been discussing this in a number of PRs and issues, so I figured it might be good to consolidate some of the thoughts. I'd also appreciate feedback from the community on which approach they like most and why they like it (which features are they anticipating using/already use), and what concerns they with this.

[flovilmart]

Great summary of the situation, we really need to make a decision 'bout it

[rogerhu]

How often do dependencies from core libs (i.e. Amazon - https://github.com/aws/aws-sdk-js, https://github.com/google/google-api-nodejs-client) would impact approach #1 vs approach #2? i.e. would you end up blocking iterations on one extension (i.e. push) needed to be updated but the other adapters (i.e . file storage) would also have to be updated? That would favor approach #2 as well.

I agree we need a standard services (i.e. Amazon SNS leverages shared code for GCM). Also, Amazon for instance you can get away with using S3 but GCM only.

[inlined]

I'm generally in favor of #2 for several reasons:

1. Customers are likely to deploy ParseServer against one *aaS, and this simplifies the real customer's life.
2. Boilerplate for authenticating or configuring a backend can be consolidated in the module require and reused in various adapter factories.
3. This is less likely than Why?? #1 to lead to either dependency bloat or dependency rot (where nobody remembers why a dependency is needed and is afraid to update it)
4. It seems plausible that providers would benefit from some network effect between their adapters (e.g. analytics integration with files or push); Add JS Syntax Highlighting to README.md code block #2 will encourage ParseServer adapters to be worth more than the sum of their parts.
5. Similarly, some Parse features were built with assumptions about the network effect (e.g. Parse Push is built atop Parse data). It's reasonable that a push adapter would assume a particular implementation of PFInstallation.
6. Not all providers will be created equal. Some may be flaky or hard to use; some will be a delight. Grouping adapters into a provider lets devs discuss the quality of AWS, Azure, GCP, and bob's super-cool software house holistically. There's fewer things to remember and devs are less likely to suffer decision fatigue.
7. Customers should be wary about mixing and matching adapters for the reasons alluded to above. It makes sense to both unit test the various adapters in a provider but to also have integration tests for the providers used as a whole. As a developer, I would feel safer knowing that I was using adapters that have been tested to work well together.

As my own devil's advocate, the only downside I can think of for #2 is that it will probably lead to a world where official modules from the top 3 IaaS companies become the dominant providers. I'm not sure I see this as a bad thing or something that wasn't already inevitable. Though I'm now a Google employee on the Firebase team, I'm also trying to act as an ex-Parse employee here. Our primary concern should be the quality and ease of use for those deploying ParseServer, not those contributing to or extending it. Option #2 seems to require the least research, boilerplate, useless dependencies, or debugging from our customers.

[felixrieseberg]

We at Microsoft already started working on implementations, and I agree with my Google colleague that #2 seems to be the most user-friendly option. We already released https://github.com/felixrieseberg/parse-server-azure-storage, but it'd be easy for providers to combine multiple sub-modules into one provider-module.

Quick example: If you're working with multiple Azure services, you use the azure npm module, which itself has a bunch of dependencies (like azure-storage). If you just need Azure storage, the end user can also just require azure-storage directly.

[felixrieseberg]

To put some code behind our support for option #2: I basically envision the way we enable people to migrate with a module like this one, which will also be useful for users as a "main entry point", allowing us to keep all the relevant information in one readme.

[asciimike]

To play devils advocate, I think that 2 is just a special case of 3 in that large providers are just third party devs with more resources, as everyone is still going to be creating their own.

To give a concrete example on how I view the benefits of 1: since we now have five (yes, five!) File Adapters (Filesystem, GridStore, S3, Azure, GCP), if someone wanted to create a new one, in a parse-server-extension-files module, it would be as easy as cloning the repo, copying an adapter and a test, modifying them, and submitting the PR. It's also likely that the adapters would stay more in sync, because it's immediately clear if an adapter breaks due to a change made, and the owners could refuse to accept changes that break other adapters to ensure quality. I have a feeling that we would also want to try and ensure regular maintainers own these codebases to prevent the issues mentioned above (package cruft, test coverage, etc.).

The alternative experience would involve finding one of the repos which implements FileAdapter, cloning it (and hoping it's got the most up to date version of the FileAdapter interface), copying it to a new repo, writing/testing the code, then setting up CI properly, and individually maintaining it (which can be a decent burden). And any time a major change occurs, notifications would be less automatic (things would break) which I think would result in more packages which are less maintained.

I do think that from a deployment perspective 2 makes much more sense, as developers are likely to stick to a single stack for familiarity, cost savings/credits, "synergy", superstition, or some combination of the above. I think it makes more sense for large providers, as maintaining the suite of products that work well with parse-server is far easier. I just worry that these modules will become more divergent and balkanized as time goes on, which is why I wanted to bring this up and have the discussion.

I'd love some "unaffiliated" developers thoughts (especially those looking to implement a module but who are blocked on process--how can we help create a framework that makes building extensions easier? Would creating provider based extensions allow you to build off of those easier?)

[OneCricketeer]

Also interested in the separation of extensions.

On the Database front, there is #20.

[ericmbarnard]

After reading this I'm thinking that 2 will come out of going down 3's road (as @felixrieseberg has shown with the example azure repo).

To continue with felix's example, I like Azure and definitely would like to use *most of their services with parse-server. That said, however, Azure has a TON of services, and I would prefer to not have a giant module with adapters for every single one of their services in it. For example, I would like to use Azure's storage service for blobs, and their DocumentDB service for general data, but I know that what I'm building will probably also need to have some data stored in a SQL database and probably even some in a cache. If a Postgres adapter comes to fruition, I'll probably use that instead of Azure SQL DB and I wouldn't want all of MSSQL's packages just sitting in my project doing nothing. The same case if I wanted to use a Redis adapter as well. I would still use Azure's platform for running Postgres and Redis, but I would assume indie devs would build those adapters instead of MS.

So I'm thinking 3 is ultimately the way to go. The big companies AND the smaller devs can put out the individual extensions/adapters and then, if they want, also keep a giant module that brings them all together to accomplish 2 and for ease of new folks to get going.

In order to support 3, I think a template or simple interfaces-only package (kind of how Passport does Strategies) would get us there fairly easily. From a visibility/discoverability standpoint - I'm not too worried that the major companies would have much trouble getting the word out. Within a day of the original bad news, I had found the major blogs and instructions for setting parse-server up on Google/MS/AWS .

[flovilmart]

I believe we can close that one as we've done it.

Orignal adapters have been ported to https://github.com/parse-server-modules/

[natario1]

@flovilmart going to write this in a casual place, thank you for your work on everything!

[asciimike]

👏 👏 👏