Skip to content

Fixed issue that prevented Postgres Tests from passing locally and on any port other than 5432 in travis #6531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 112 commits into from
Apr 3, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
841768c
Attempting to fix Postgres issue
cbaker6 Mar 14, 2020
33da65e
Attempting to fix Postgres issue
cbaker6 Mar 14, 2020
43b714d
Attempting to fix Postgres
cbaker6 Mar 14, 2020
8010633
Attempting to fix Postgres issue
cbaker6 Mar 14, 2020
cc2506a
Attempting to fix postgres
cbaker6 Mar 14, 2020
05181a7
Attempting to fix postgres
cbaker6 Mar 14, 2020
1ff769d
Attempting to fix postgres
cbaker6 Mar 14, 2020
fba1b83
Attempting to fix postgres
cbaker6 Mar 14, 2020
edf60e8
Attempting to fix postgres
cbaker6 Mar 14, 2020
c884e5d
Attempting to fix postgres
cbaker6 Mar 14, 2020
e553200
Attempting to fix postgres
cbaker6 Mar 14, 2020
c7b4411
Attempting to fix postgres
cbaker6 Mar 14, 2020
063dd2f
Attempting to fix postgres
cbaker6 Mar 14, 2020
d357a47
Attempting to fix postgres
cbaker6 Mar 14, 2020
b048966
Attempting to fix postgres
cbaker6 Mar 14, 2020
299b91d
Attempting to fix postgres
cbaker6 Mar 14, 2020
6adf026
Attempting to fix postgres
cbaker6 Mar 14, 2020
87e37a1
Attempting to fix postgres
cbaker6 Mar 14, 2020
c20a82e
Attempting to fix postgres
cbaker6 Mar 14, 2020
76a29d0
Attempting to fix postgres
cbaker6 Mar 14, 2020
b90abc4
Update .travis.yml
cbaker6 Mar 14, 2020
5d4019d
Attempting to fix postgres
cbaker6 Mar 14, 2020
028daee
Attempting to fix postgres
cbaker6 Mar 14, 2020
387258c
Attempting to fix postgres
cbaker6 Mar 14, 2020
dc3afa6
Attempting to fix postgres
cbaker6 Mar 14, 2020
06045a5
Attempting to fix postgres
cbaker6 Mar 14, 2020
ebcbbb0
Attempting to fix postgres
cbaker6 Mar 14, 2020
72f72a1
Attempting to fix postgres
cbaker6 Mar 14, 2020
2047e2e
Attempting to fix postgres
cbaker6 Mar 14, 2020
fa10637
Attempting to fix postgres
cbaker6 Mar 14, 2020
0bdc4f4
Temporary fix: separated mongo and postgres in travis
cbaker6 Mar 14, 2020
d60203d
Allowing all postgres to fail
cbaker6 Mar 14, 2020
a7c964b
Allowing multiple names to fail
cbaker6 Mar 14, 2020
d3e795a
Removing preinstalled versions of postgres from list
cbaker6 Mar 15, 2020
b0e224f
Use postgres made for dist
cbaker6 Mar 15, 2020
98d83e9
Second round
cbaker6 Mar 15, 2020
ddd6f4a
Round 3
cbaker6 Mar 15, 2020
efa17ff
Round 4
cbaker6 Mar 15, 2020
6a98945
Round 5
cbaker6 Mar 15, 2020
a3f72b8
Fixed issue with random postgres fail
cbaker6 Mar 15, 2020
5999aff
Added back postgres 11 and 12 to allow_failures
cbaker6 Mar 15, 2020
579124c
added travis scripts for postgres
cbaker6 Mar 17, 2020
38a2b9e
Setting up before_install and before_script
cbaker6 Mar 17, 2020
8b0f875
make scripts executable
cbaker6 Mar 17, 2020
ac3be67
Merge branch 'master' of https://github.com/netreconlab/parse-server
cbaker6 Mar 17, 2020
0052963
Update .travis.yml
cbaker6 Mar 17, 2020
a375604
add sourcing in script
cbaker6 Mar 17, 2020
8d26cbc
trying to fix source
cbaker6 Mar 17, 2020
dfcbe0d
fixing env var in script
cbaker6 Mar 17, 2020
a5aa417
fixed ; near then
cbaker6 Mar 17, 2020
cc91dc4
Cleaning up travis file
cbaker6 Mar 17, 2020
0e33072
Finishing clean up
cbaker6 Mar 17, 2020
773ca67
Fixing allow_failures since "name" was removed
cbaker6 Mar 17, 2020
fe0077c
Update .travis.yml
cbaker6 Mar 17, 2020
559e0a7
Removed Postgres 11 from allow_failures
cbaker6 Mar 17, 2020
2ceff98
I think using travis default postgres port of 5433 will allow us to n…
cbaker6 Mar 17, 2020
57a81cb
Merge branch 'master' of https://github.com/netreconlab/parse-server
cbaker6 Mar 17, 2020
7776457
Switching travis to postgres port 5433
cbaker6 Mar 17, 2020
59a3e8a
modifying script for test
cbaker6 Mar 17, 2020
3a099c1
modifying script for test
cbaker6 Mar 17, 2020
fc2006d
modifying script for test
cbaker6 Mar 17, 2020
4ffee37
reverting back to working way with removing postgres from image
cbaker6 Mar 17, 2020
8d17b04
Reverted back to removing postgres from image
cbaker6 Mar 17, 2020
84b4a64
removing postgres 12
cbaker6 Mar 18, 2020
839d3e6
removed postgres-12 from allow_failures
cbaker6 Mar 19, 2020
c1173cb
updated postgres method from deprecated. Also updating postgis to 3.0
cbaker6 Mar 23, 2020
648ea74
updated postgis to 3.0
cbaker6 Mar 23, 2020
3842083
Merge branch 'master' into master
cbaker6 Mar 23, 2020
6a6c2d4
Update .travis.yml
cbaker6 Mar 23, 2020
fca6973
fix Postgres test issues that causes some tests to fail locally when …
cbaker6 Mar 23, 2020
cc0f9c7
Merge branch 'master' of https://github.com/netreconlab/parse-server
cbaker6 Mar 23, 2020
bcf2116
removed dependence on creating adapter locally in testcase and use co…
cbaker6 Mar 24, 2020
530bc9d
fixed one more potential issue with postgres adapter
cbaker6 Mar 24, 2020
b4c6ee0
modified travis and script to leverage PARSE_SERVER_TEST_DATABASE_URI
cbaker6 Mar 25, 2020
b8ee28e
Merge branch 'master' into master
cbaker6 Mar 25, 2020
bdac2f4
changes to before install to reduce time
cbaker6 Mar 25, 2020
e90eeae
Merge branch 'master' of https://github.com/netreconlab/parse-server
cbaker6 Mar 25, 2020
4192af3
add updated port to script for psql connection
cbaker6 Mar 25, 2020
48d1461
fixed arg for psql
cbaker6 Mar 25, 2020
5265115
added back conf file as it probably contains the default permissions
cbaker6 Mar 25, 2020
4cd60df
seeing if stopping/starting postgres back-to-back was the issue
cbaker6 Mar 25, 2020
183e273
checking fix
cbaker6 Mar 25, 2020
5457858
revert to working version of install script
cbaker6 Mar 25, 2020
5a175d5
leave default port
cbaker6 Mar 25, 2020
2455281
removing uninstall of postgres from script
cbaker6 Mar 25, 2020
36d590a
looks like postgres won't start until the others are removed
cbaker6 Mar 25, 2020
ee57800
forgoing using posgres service and starting directly, hoping it allow…
cbaker6 Mar 25, 2020
8300f20
removed sudo for starting postgres
cbaker6 Mar 25, 2020
dc6e595
adding back sude, but using default user for postgres
cbaker6 Mar 25, 2020
59c9031
fix
cbaker6 Mar 25, 2020
8e27b46
don't start service
cbaker6 Mar 25, 2020
268c08a
init db cluster before starting
cbaker6 Mar 25, 2020
f93620f
remove folder befor init
cbaker6 Mar 25, 2020
c1dde02
remove folder befor init
cbaker6 Mar 25, 2020
b269e6b
still have to stop service
cbaker6 Mar 25, 2020
bd5a1f8
switch ports after new configs are made
cbaker6 Mar 25, 2020
160f5a5
switch ports after new configs are made
cbaker6 Mar 25, 2020
2eb7987
went back to removing to get it to start properly
cbaker6 Mar 25, 2020
f635706
seeing if you can use postgres installaition out of the box
cbaker6 Mar 25, 2020
fff3f0d
seeing if you can use postgres installaition out of the box
cbaker6 Mar 25, 2020
f03310e
copy hba file, then stop,start
cbaker6 Mar 26, 2020
020080e
testing using port as argument
cbaker6 Mar 26, 2020
79c7949
working version
cbaker6 Mar 26, 2020
c27c304
round 2
cbaker6 Mar 26, 2020
bb86193
updated documentation to reflect changes
cbaker6 Mar 26, 2020
4a8650e
use restart instead of stop/start for postgres
cbaker6 Mar 27, 2020
a70e1f9
removed extra lines to ensure consistency with future builds
cbaker6 Mar 27, 2020
9073081
reload instead of restart postgres
cbaker6 Mar 28, 2020
a03d511
restart postgres since we can't currently connect
cbaker6 Mar 28, 2020
fc73f17
Switch to testonly for postgres
cbaker6 Apr 3, 2020
861f048
run testonly
cbaker6 Apr 3, 2020
8d40f0c
Declare as mongo test since it's handed coded to a mongo uri
cbaker6 Apr 3, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
packages:
- postgresql-11-postgis-3
- postgresql-11-postgis-3-scripts
env: POSTGRES_MAJOR_VERSION=11 PARSE_SERVER_TEST_DB=postgres
env: POSTGRES_MAJOR_VERSION=11 PARSE_SERVER_TEST_DB=postgres PARSE_SERVER_TEST_DATABASE_URI=postgres://localhost:5433/parse_server_postgres_adapter_test_database
before_install: bash scripts/before_install_postgres.sh
before_script: bash scripts/before_script_postgres.sh
- stage: release
Expand Down
7 changes: 6 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,18 @@ Once you have babel running in watch mode, you can start making changes to parse

If your pull request introduces a change that may affect the storage or retrieval of objects, you may want to make sure it plays nice with Postgres.

* Run the tests against the postgres database with `PARSE_SERVER_TEST_DB=postgres npm test`. You'll need to have postgres running on your machine and setup [appropriately](https://github.com/parse-community/parse-server/blob/master/.travis.yml#L37) or use [`Docker`](#run-a-parse-postgres-with-docker).
* Run the tests against the postgres database with `PARSE_SERVER_TEST_DB=postgres PARSE_SERVER_TEST_DATABASE_URI=postgres://postgres:password@localhost:5432/parse_server_postgres_adapter_test_database npm run testonly`. You'll need to have postgres running on your machine and setup [appropriately](https://github.com/parse-community/parse-server/blob/master/.travis.yml#L43) or use [`Docker`](#run-a-parse-postgres-with-docker).
* The Postgres adapter has a special debugger that traces all the sql commands. You can enable it with setting the environment variable `PARSE_SERVER_LOG_LEVEL=debug`
* If your feature is intended to only work with MongoDB, you should disable PostgreSQL-specific tests with:

- `describe_only_db('mongo')` // will create a `describe` that runs only on mongoDB
- `it_only_db('mongo')` // will make a test that only runs on mongo
- `it_exclude_dbs(['postgres'])` // will make a test that runs against all DB's but postgres
* Similarly, if your feature is intended to only work with PostgreSQL, you should disable MongoDB-specific tests with:

- `describe_only_db('postgres')` // will create a `describe` that runs only on postgres
- `it_only_db('postgres')` // will make a test that only runs on postgres
- `it_exclude_dbs(['mongo'])` // will make a test that runs against all DB's but mongo

#### Run a Parse Postgres with Docker

Expand Down
23 changes: 8 additions & 15 deletions scripts/before_install_postgres.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,28 @@

set -e

source ~/.nvm/nvm.sh

echo "[SCRIPT] Before Install Script :: Setup Postgres ${POSTGRES_MAJOR_VERSION}"

nvm install $NODE_VERSION
nvm use $NODE_VERSION
npm install -g greenkeeper-lockfile@1

sudo sed -i 's/port = 5433/port = 5432/' /etc/postgresql/${POSTGRES_MAJOR_VERSION}/main/postgresql.conf

if [[ $POSTGRES_MAJOR_VERSION -lt 11 ]]; then
# Setup postgres 9 or 10
sudo sed -i 's/port = 5432/port = 5433/' /etc/postgresql/${POSTGRES_MAJOR_VERSION}/main/postgresql.conf

# Stop the current running service
sudo service postgresql stop

# Remove correct version of postgres
if [[ $POSTGRES_MAJOR_VERSION -lt 10 ]]; then
sudo apt-get remove -q 'postgresql-10.*'
else
sudo apt-get remove -q 'postgresql-9.*'
fi

sudo service postgresql start ${POSTGRES_MAJOR_VERSION}
sudo service postgresql start

else

# Setup postgres 11 or higher

#Copy defauilt hba config file and tell postgres to restart
sudo cp /etc/postgresql/{10,${POSTGRES_MAJOR_VERSION}}/main/pg_hba.conf
sudo service postgresql stop
# Remove previous versions of postgres
sudo apt-get remove -q 'postgresql-9.*' 'postgresql-10.*'
sudo service postgresql start ${POSTGRES_MAJOR_VERSION}
sudo systemctl restart postgresql@${POSTGRES_MAJOR_VERSION}-main
fi
4 changes: 1 addition & 3 deletions scripts/before_script_postgres.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ set -e

echo "[SCRIPT] Before Script :: Setup Parse DB for Postgres ${POSTGRES_MAJOR_VERSION}"

node -e 'require("./lib/index.js")'

psql -v ON_ERROR_STOP=1 --username "postgres" --dbname "${POSTGRES_DB}" <<-EOSQL
psql -v ON_ERROR_STOP=1 -p 5433 --username "postgres" --dbname "${POSTGRES_DB}" <<-EOSQL
CREATE DATABASE parse_server_postgres_adapter_test_database;
\c parse_server_postgres_adapter_test_database;
CREATE EXTENSION postgis;
Expand Down
2 changes: 1 addition & 1 deletion spec/GridFSBucketStorageAdapter.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ async function expectMissingFile(gfsAdapter, name) {
}
}

describe('GridFSBucket and GridStore interop', () => {
describe_only_db('mongo')('GridFSBucket and GridStore interop', () => {
beforeEach(async () => {
const gsAdapter = new GridStoreAdapter(databaseURI);
const db = await gsAdapter._connect();
Expand Down
19 changes: 4 additions & 15 deletions spec/ParseQuery.FullTextSearch.spec.js
Original file line number Diff line number Diff line change
@@ -1,25 +1,14 @@
'use strict';

const MongoStorageAdapter = require('../lib/Adapters/Storage/Mongo/MongoStorageAdapter')
.default;
const mongoURI =
'mongodb://localhost:27017/parseServerMongoAdapterTestDatabase';
const PostgresStorageAdapter = require('../lib/Adapters/Storage/Postgres/PostgresStorageAdapter')
.default;
const postgresURI =
'postgres://localhost:5432/parse_server_postgres_adapter_test_database';
const Config = require('../lib/Config');
const Parse = require('parse/node');
const request = require('../lib/request');
let databaseAdapter;

const fullTextHelper = () => {
if (process.env.PARSE_SERVER_TEST_DB === 'postgres') {
if (!databaseAdapter) {
databaseAdapter = new PostgresStorageAdapter({ uri: postgresURI });
}
} else {
databaseAdapter = new MongoStorageAdapter({ uri: mongoURI });
}
const config = Config.get('test');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary. Have you read the Contributor Guide

Does PARSE_SERVER_TEST_DB=postgres npm test work for you?

Copy link
Contributor Author

@cbaker6 cbaker6 Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dplewis this test fails locally on my system using that command if I don’t have Postgres running on localhost using the standard port, 5432. This fix makes it work on any address/port Postgres is binded to by using the already defined PARSE_SERVER_TEST_DATABASE_URI environment variable. In the test cases current form, PARSE_SERVER_TEST_DATABASE_URI doesn’t work as documented intended without this fix. Two test fail in this case, which is I only fixed those.

You can replicate the failure by dowloading Postgres via docker and binding to a different port.

I suspect this is why the Travis file has to change the Postgres config to 5432 as the newly installed Postgres installations use 5433 and the PARSE_SERVER_TEST_DATABASE_URI doesn’t work properly Edit: because of ParseQuery.FullTextSearch.spec.js and PostgresInitOptions.spec.js (the only two tests that fail) has the URI for postgres/mongo hardcoded instead of leveraging the config file like other tests (see my initial fix).

Copy link
Contributor Author

@cbaker6 cbaker6 Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my last point, after this pull request is reconciled, I plan on creating another one which edits the travis file and scripts to not have to stop/start postgres, nor have to remove older versions of postgres that are running on port 5432. Using PARSE_SERVER_TEST_DATABASE_URI should allow us to test on port 5433 directly after install of postgres without changing the image postgres config files (hopefully). I'm assuming this will speed up the postgres builds along with keeping the fix of the old postgres build issues.

If you think it's relevant enough, I can make the changes here if you think they are related enough and we can see how they go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, PARSE_SERVER_TEST_DATABASE_URI isn't mentioned in the Contributors Guide, but it is in the spec/helper.js and spec/ParseServer.js files. I can add this in as well.

Copy link
Contributor Author

@cbaker6 cbaker6 Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lastly, if anyone attempts to run a local test using the official Postgres (which PostGIS docker image uses), POSTGRES_PASSWORD has to be set or else the image won't build. This means that all local postgres tests using docker will require the use if PARSE_SERVER_TEST_DATABASE_URI as the default postgres URI set in the spec won't work. See my example below:

docker run -d --name parse-postgres -p 5432:5432 -e POSTGRES_USER=$USER -e POSTGRES_PASSWORD=password --rm postgis/postgis:11-2.5-alpine && sleep 20 && docker exec -it parse-postgres psql -U $USER -c 'CREATE DATABASE parse_server_postgres_adapter_test_database;' && docker exec -it parse-postgres psql -U $USER -c 'CREATE EXTENSION postgis;' -d parse_server_postgres_adapter_test_database && docker exec -it parse-postgres psql -U $USER -c 'CREATE EXTENSION postgis_topology;' -d parse_server_postgres_adapter_test_database
...
PARSE_SERVER_TEST_DB=postgres PARSE_SERVER_TEST_DATABASE_URI=postgres://$USER:password@localhost:5432/parse_server_postgres_adapter_test_database npm test

Note that I didn't use the mdillion/postgis image discussed in the Contributors Guide as it hasn't been updated in over a year.

databaseAdapter = config.database.adapter;

const subjects = [
'coffee',
'Coffee Shopping',
Expand Down
1 change: 1 addition & 0 deletions spec/PostgresInitOptions.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ const Parse = require('parse/node').Parse;
const PostgresStorageAdapter = require('../lib/Adapters/Storage/Postgres/PostgresStorageAdapter')
.default;
const postgresURI =
process.env.PARSE_SERVER_TEST_DATABASE_URI ||
'postgres://localhost:5432/parse_server_postgres_adapter_test_database';
const ParseServer = require('../lib/index');
const express = require('express');
Expand Down
1 change: 1 addition & 0 deletions spec/PostgresStorageAdapter.spec.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
const PostgresStorageAdapter = require('../lib/Adapters/Storage/Postgres/PostgresStorageAdapter')
.default;
const databaseURI =
process.env.PARSE_SERVER_TEST_DATABASE_URI ||
'postgres://localhost:5432/parse_server_postgres_adapter_test_database';
const Config = require('../lib/Config');

Expand Down