Skip to content

Improved match aggregate #4495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 20, 2018
Merged

Conversation

dplewis
Copy link
Member

@dplewis dplewis commented Jan 11, 2018

  • Pointer transform
  • ObjectId transform (do we want to allow both _id and objectId?)
  • Allows for matching all field names (PG)
  • Multiple Comparison (PG)
  • Complex matching (PG)
  • Support for $or (PG)
  • Refactored aggregate to use pg-promise query formatting

Edit: Should I replace the REST calls with query.aggregate() in the tests?

@codecov
Copy link

codecov bot commented Jan 11, 2018

Codecov Report

Merging #4495 into master will increase coverage by 0.05%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4495      +/-   ##
==========================================
+ Coverage    92.8%   92.85%   +0.05%     
==========================================
  Files         118      118              
  Lines        8394     8442      +48     
==========================================
+ Hits         7790     7839      +49     
+ Misses        604      603       -1
Impacted Files Coverage Δ
...dapters/Storage/Postgres/PostgresStorageAdapter.js 97.12% <100%> (ø) ⬆️
src/Adapters/Storage/Mongo/MongoStorageAdapter.js 96.06% <100%> (+0.13%) ⬆️
src/RestWrite.js 93.46% <0%> (ø) ⬆️
src/Adapters/Cache/InMemoryCache.js 100% <0%> (+8.33%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33890bb...40f4ba0. Read the comment docs.

@dplewis dplewis requested a review from flovilmart January 11, 2018 19:31
@dplewis dplewis requested a review from vitaly-t January 12, 2018 02:34
@dplewis dplewis requested a review from montymxb January 15, 2018 22:51
Copy link
Contributor

@montymxb montymxb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small nits that don't necessarily need to be changed.

Beyond that the actual PG related code, according to the tests, seems to be good but I'll defer to someone for a proper analysis of that code.

You're probably ok with using the rest calls, ultimately so long as it still runs the same functionality internally.


// Match view { $gt: 750, $lt: 850 }
expect(resp.results.some(result => result.score === 10)).toEqual(true);
expect(resp.results.some(result => result.views === 800)).toEqual(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could just check that the value is equal to the expected value directly, rather than an equality check and bool check.

Copy link
Member Author

@dplewis dplewis Jan 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mongo and PG return different orders for results

If I check directly, one of the test would fail even tho they return correct results.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's the case wouldn't this fail anyways with a comparison against a different result?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.some() returns true if at least one element matches. So the order doesn’t matter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also helps if future databases changes matches internally or query optimization returns different results order

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, gotcha, alright we're good then.

expect(results[0].pointer.objectId).toEqual(pointer1.id);
expect(results[1].pointer.objectId).toEqual(pointer1.id);
expect(results.some(result => result.objectId === obj1.id)).toEqual(true);
expect(results.some(result => result.objectId === obj3.id)).toEqual(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, you can just compare that the expected objectId matches directly

@dplewis
Copy link
Member Author

dplewis commented Jan 20, 2018

@flovilmart I know you are a busy man. Can you look this over? 🙏

Copy link
Contributor

@flovilmart flovilmart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s looking good, but the _id/objectId transformation is confusing as they are never object ID’s. It’s the primary key for the pipeline, so I guess we should keep it as _id.
However, when a user references the objectId key by putting $objectId, this should be transformed to $_id

@dplewis
Copy link
Member Author

dplewis commented Jan 20, 2018

@flovilmart I agree that can put either for the input but the output is already objectId

@dplewis dplewis merged commit 64e568d into parse-community:master Jan 20, 2018
@dplewis dplewis deleted the match-pointer branch January 20, 2018 14:00
@vitaly-t
Copy link
Contributor

vitaly-t commented Jan 21, 2018

@dplewis @flovilmart

Hi guys! Although presently I do not have time to be doing any support here, much less PR-s, I will just add that this library can perform better and have much simpler code, if it starts using the following important features of the pg-promise driver:

  • multi-result methods multi and multiResult - offer a great performance improvement, allowing to execute multiple queries and get all results with a single IO operation.
  • nested named parameters - can simplify the formatting logic a lot. Right now the library does way too much with the basic $1, $2,... variables because it ignores the power of nested named parameters that's available to it.
  • Custom Type Formatting, makes it so much simpler to customize your formatting, which is even more flexible when used with the nested named parameters.

UnderratedDev pushed a commit to UnderratedDev/parse-server that referenced this pull request Mar 21, 2020
* Improve aggregate queries

* using pg-promise query formatting

* match multiple comparison

* $or and complex match
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants