Skip to content

Commit 7e868b2

Browse files
drew-grossdrew-gross
authored
Unique indexes (#1971)
* Add unique indexing * Add unique indexing for username/email * WIP * Finish unique indexes * Notes on how to upgrade to 2.3.0 safely * index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (#1922)" * reconfigure username/email tests * Start dealing with test shittyness * Remove tests for files that we are removing * most tests passing * fix failing test * Make specific server config for tests async * Fix more tests * fix more tests * Fix another test * fix more tests * Fix email validation * move some stuff around * Destroy server to ensure all connections are gone * Fix broken cloud code * Save callback to variable * no need to delete non existant cloud * undo * Fix all tests where connections are left open after server closes. * Fix issues caused by missing gridstore adapter * Update guide for 2.3.0 and fix final tests * use strict * don't use features that won't work in node 4 * Fix syntax error * Fix typos * Add duplicate finding command * Update 2.3.0.md
1 parent 6415a35 commit 7e868b2

37 files changed

+1738
-1528
lines changed

‎2.3.0.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Upgrading Parse Server to version 2.3.0
2+
3+
Parse Server version 2.3.0 begins using unique indexes to ensure User's username and email are unique. This is not a backwards incompatable change, but it may in some cases cause a significant performance regression until the index finishes building. Building the unique index before upgrading your Parse Server version will eliminate the performance impact, and is a recommended step before upgrading any app to Parse Server 2.3.0. New apps starting with version 2.3.0 do not need to take any steps before beginning their project.
4+
5+
If you are using MongoDB in Cluster or Replica Set mode, we recommend reading Mongo's [documentation on index building](https://docs.mongodb.com/v3.0/tutorial/build-indexes-on-replica-sets/) first. If you are not using these features, you can execute the following commands from the Mongo shell to build the unique index. You may also want to create a backup first.
6+
7+
```js
8+
// Select the database that your Parse App uses
9+
use parse;
10+
11+
// Select the collection your Parse App uses for users. For migrated apps, this probably includes a collectionPrefix.
12+
var coll = db['your_prefix:_User'];
13+
14+
// You can check if the indexes already exists by running coll.getIndexes()
15+
coll.getIndexes();
16+
17+
// The indexes you want should look like this. If they already exist, you can skip creating them.
18+
{
19+
"v" : 1,
20+
"unique" : true,
21+
"key" : {
22+
"username" : 1
23+
},
24+
"name" : "username_1",
25+
"ns" : "parse.your_prefix:_User",
26+
"background" : true,
27+
"sparse" : true
28+
}
29+
30+
{
31+
"v" : 1,
32+
"unique" : true,
33+
"key" : {
34+
"email" : 1
35+
},
36+
"name" : "email_1",
37+
"ns" : "parse.your_prefix:_User",
38+
"background" : true,
39+
"sparse" : true
40+
}
41+
42+
// Create the username index.
43+
// "background: true" is mandatory and avoids downtime while the index builds.
44+
// "sparse: true" is also mandatory because Parse Server uses sparse indexes.
45+
coll.ensureIndex({ username: 1 }, { background: true, unique: true, sparse: true });
46+
47+
// Create the email index.
48+
// "background: true" is still mandatory.
49+
// "sparse: true" is also mandatory both because Parse Server uses sparse indexes, and because email addresses are not required by the Parse API.
50+
coll.ensureIndex({ email: 1 }, { background: true, unique: true, sparse: true });
51+
```
52+
53+
There are some issues you may run into during this process:
54+
55+
## Mongo complains that the index already exists, but with different options
56+
57+
In this case, you will need to remove the incorrect index. If your app relies on the existence of the index in order to be performant, you can create a new index, with "-1" for the direction of the field, so that it counts as different options. Then, drop the conflicting index, and create the unique index.
58+
59+
## There is already non-unique data in the username or email field
60+
61+
This is possible if you have explicitly set some user's emails to null. If this is bogus data, and those null fields shoud be unset, you can unset the null emails with this command. If your app relies on the difference between null and unset emails, you will need to upgrade your app to treat null and unset emails the same before building the index and upgrading to Parse Server 2.3.0.
62+
63+
```js
64+
coll.update({ email: { $exists: true, $eq: null } }, { $unset: { email: '' } }, { multi: true })
65+
```
66+
67+
## There is already non-unique data in the username or email field, and it's not nulls
68+
69+
This is possible due to a race condition in previous versions of Parse Server. If you have this problem, it is unlikely that you have a lot of rows with duplicate data. We recommend you clean up the data manually, by removing or modifying the offending rows.
70+
71+
This command, can be used to find the duplicate data:
72+
73+
```js
74+
coll.aggregate([
75+
{$match: {"username": {"$ne": null}}},
76+
{$group: {_id: "$username", uniqueIds: {$addToSet: "$_id"}, count: {$sum: 1}}},
77+
{$match: {count: {"$gt": 1}}},
78+
{$project: {id: "$uniqueIds", username: "$_id", _id : 0} },
79+
{$unwind: "$id" },
80+
{$out: '_duplicates'} // Save the list of duplicates to a new, "_duplicates collection. Remove this line to just output the list.
81+
], {allowDiskUse:true})
82+
```

‎package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"parse-server-simple-mailgun-adapter": "^1.0.0",
4141
"redis": "^2.5.0-1",
4242
"request": "^2.65.0",
43+
"request-promise": "^3.0.0",
4344
"tv4": "^1.2.7",
4445
"winston": "^2.1.1",
4546
"winston-daily-rotate-file": "^1.0.1",

‎spec/CloudCode.spec.js

Lines changed: 76 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,28 @@
11
"use strict"
22
const Parse = require("parse/node");
33
const request = require('request');
4+
const rp = require('request-promise');
45
const InMemoryCacheAdapter = require('../src/Adapters/Cache/InMemoryCacheAdapter').InMemoryCacheAdapter;
56

67
describe('Cloud Code', () => {
78
it('can load absolute cloud code file', done => {
8-
setServerConfiguration({
9-
serverURL: 'http://localhost:8378/1',
10-
appId: 'test',
11-
masterKey: 'test',
12-
cloud: __dirname + '/cloud/cloudCodeRelativeFile.js'
13-
});
14-
Parse.Cloud.run('cloudCodeInFile', {}, result => {
15-
expect(result).toEqual('It is possible to define cloud code in a file.');
16-
done();
17-
});
9+
reconfigureServer({ cloud: __dirname + '/cloud/cloudCodeRelativeFile.js' })
10+
.then(() => {
11+
Parse.Cloud.run('cloudCodeInFile', {}, result => {
12+
expect(result).toEqual('It is possible to define cloud code in a file.');
13+
done();
14+
});
15+
})
1816
});
1917

2018
it('can load relative cloud code file', done => {
21-
setServerConfiguration({
22-
serverURL: 'http://localhost:8378/1',
23-
appId: 'test',
24-
masterKey: 'test',
25-
cloud: './spec/cloud/cloudCodeAbsoluteFile.js'
26-
});
27-
Parse.Cloud.run('cloudCodeInFile', {}, result => {
28-
expect(result).toEqual('It is possible to define cloud code in a file.');
29-
done();
30-
});
19+
reconfigureServer({ cloud: './spec/cloud/cloudCodeAbsoluteFile.js' })
20+
.then(() => {
21+
Parse.Cloud.run('cloudCodeInFile', {}, result => {
22+
expect(result).toEqual('It is possible to define cloud code in a file.');
23+
done();
24+
});
25+
})
3126
});
3227

3328
it('can create functions', done => {
@@ -568,67 +563,75 @@ describe('Cloud Code', () => {
568563
});
569564

570565
it('clears out the user cache for all sessions when the user is changed', done => {
566+
let session1;
567+
let session2;
568+
let user;
571569
const cacheAdapter = new InMemoryCacheAdapter({ ttl: 100000000 });
572-
setServerConfiguration(Object.assign({}, defaultConfiguration, { cacheAdapter: cacheAdapter }));
573-
Parse.Cloud.define('checkStaleUser', (request, response) => {
574-
response.success(request.user.get('data'));
575-
});
570+
reconfigureServer({ cacheAdapter })
571+
.then(() => {
572+
Parse.Cloud.define('checkStaleUser', (request, response) => {
573+
response.success(request.user.get('data'));
574+
});
576575

577-
let user = new Parse.User();
578-
user.set('username', 'test');
579-
user.set('password', 'moon-y');
580-
user.set('data', 'first data');
581-
user.signUp()
576+
user = new Parse.User();
577+
user.set('username', 'test');
578+
user.set('password', 'moon-y');
579+
user.set('data', 'first data');
580+
return user.signUp();
581+
})
582582
.then(user => {
583-
let session1 = user.getSessionToken();
584-
request.get({
585-
url: 'http://localhost:8378/1/login?username=test&password=moon-y',
583+
session1 = user.getSessionToken();
584+
return rp({
585+
uri: 'http://localhost:8378/1/login?username=test&password=moon-y',
586586
json: true,
587587
headers: {
588588
'X-Parse-Application-Id': 'test',
589589
'X-Parse-REST-API-Key': 'rest',
590590
},
591-
}, (error, response, body) => {
592-
let session2 = body.sessionToken;
593-
594-
//Ensure both session tokens are in the cache
595-
Parse.Cloud.run('checkStaleUser')
596-
.then(() => {
597-
request.post({
598-
url: 'http://localhost:8378/1/functions/checkStaleUser',
599-
json: true,
600-
headers: {
601-
'X-Parse-Application-Id': 'test',
602-
'X-Parse-REST-API-Key': 'rest',
603-
'X-Parse-Session-Token': session2,
604-
}
605-
}, (error, response, body) => {
606-
Parse.Promise.all([cacheAdapter.get('test:user:' + session1), cacheAdapter.get('test:user:' + session2)])
607-
.then(cachedVals => {
608-
expect(cachedVals[0].objectId).toEqual(user.id);
609-
expect(cachedVals[1].objectId).toEqual(user.id);
610-
611-
//Change with session 1 and then read with session 2.
612-
user.set('data', 'second data');
613-
user.save()
614-
.then(() => {
615-
request.post({
616-
url: 'http://localhost:8378/1/functions/checkStaleUser',
617-
json: true,
618-
headers: {
619-
'X-Parse-Application-Id': 'test',
620-
'X-Parse-REST-API-Key': 'rest',
621-
'X-Parse-Session-Token': session2,
622-
}
623-
}, (error, response, body) => {
624-
expect(body.result).toEqual('second data');
625-
done();
626-
})
627-
});
628-
});
629-
});
630-
});
631-
});
591+
})
592+
})
593+
.then(body => {
594+
session2 = body.sessionToken;
595+
596+
//Ensure both session tokens are in the cache
597+
return Parse.Cloud.run('checkStaleUser')
598+
})
599+
.then(() => rp({
600+
method: 'POST',
601+
uri: 'http://localhost:8378/1/functions/checkStaleUser',
602+
json: true,
603+
headers: {
604+
'X-Parse-Application-Id': 'test',
605+
'X-Parse-REST-API-Key': 'rest',
606+
'X-Parse-Session-Token': session2,
607+
}
608+
}))
609+
.then(() => Parse.Promise.all([cacheAdapter.get('test:user:' + session1), cacheAdapter.get('test:user:' + session2)]))
610+
.then(cachedVals => {
611+
expect(cachedVals[0].objectId).toEqual(user.id);
612+
expect(cachedVals[1].objectId).toEqual(user.id);
613+
614+
//Change with session 1 and then read with session 2.
615+
user.set('data', 'second data');
616+
return user.save()
617+
})
618+
.then(() => rp({
619+
method: 'POST',
620+
uri: 'http://localhost:8378/1/functions/checkStaleUser',
621+
json: true,
622+
headers: {
623+
'X-Parse-Application-Id': 'test',
624+
'X-Parse-REST-API-Key': 'rest',
625+
'X-Parse-Session-Token': session2,
626+
}
627+
}))
628+
.then(body => {
629+
expect(body.result).toEqual('second data');
630+
done();
631+
})
632+
.catch(error => {
633+
fail(JSON.stringify(error));
634+
done();
632635
});
633636
});
634637

‎spec/DatabaseAdapter.spec.js

Lines changed: 0 additions & 23 deletions
This file was deleted.

‎spec/DatabaseController.spec.js

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎spec/FileLoggerAdapter.spec.js

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,9 @@ describe('error logs', () => {
5252
describe('verbose logs', () => {
5353

5454
it("mask sensitive information in _User class", (done) => {
55-
let customConfig = Object.assign({}, defaultConfiguration, {verbose: true});
56-
setServerConfiguration(customConfig);
57-
createTestUser().then(() => {
55+
reconfigureServer({ verbose: true })
56+
.then(() => createTestUser())
57+
.then(() => {
5858
let fileLoggerAdapter = new FileLoggerAdapter();
5959
return fileLoggerAdapter.query({
6060
from: new Date(Date.now() - 500),

‎spec/Parse.Push.spec.js

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -28,26 +28,27 @@ describe('Parse.Push', () => {
2828
}
2929
}
3030

31-
setServerConfiguration({
31+
return reconfigureServer({
3232
appId: Parse.applicationId,
3333
masterKey: Parse.masterKey,
3434
serverURL: Parse.serverURL,
3535
push: {
3636
adapter: pushAdapter
3737
}
38+
})
39+
.then(() => {
40+
var installations = [];
41+
while(installations.length != 10) {
42+
var installation = new Parse.Object("_Installation");
43+
installation.set("installationId", "installation_"+installations.length);
44+
installation.set("deviceToken","device_token_"+installations.length)
45+
installation.set("badge", installations.length);
46+
installation.set("originalBadge", installations.length);
47+
installation.set("deviceType", "ios");
48+
installations.push(installation);
49+
}
50+
return Parse.Object.saveAll(installations);
3851
});
39-
40-
var installations = [];
41-
while(installations.length != 10) {
42-
var installation = new Parse.Object("_Installation");
43-
installation.set("installationId", "installation_"+installations.length);
44-
installation.set("deviceToken","device_token_"+installations.length)
45-
installation.set("badge", installations.length);
46-
installation.set("originalBadge", installations.length);
47-
installation.set("deviceType", "ios");
48-
installations.push(installation);
49-
}
50-
return Parse.Object.saveAll(installations);
5152
}
5253

5354
it('should properly send push', (done) => {
@@ -110,7 +111,7 @@ describe('Parse.Push', () => {
110111
'X-Parse-Application-Id': 'test',
111112
},
112113
}, (error, response, body) => {
113-
expect(body.results.length).toEqual(0);
114+
expect(body.error).toEqual('unauthorized');
114115
done();
115116
});
116117
});

0 commit comments

Comments
 (0)