-
Notifications
You must be signed in to change notification settings - Fork 1.1k
PYTHON-3461 Test FaaS (AWS Lambda) Behavior Per Driver #1310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 22 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
525e29b
PYTHON-3461 Test FaaS (AWS Lambda) Behavior Per Driver
blink1073 0e82ac1
lint
blink1073 fae67e2
add test and EG config
blink1073 8220e3b
clean up
blink1073 a35f4e5
get the function working locally
blink1073 8d0e455
add missing setup
blink1073 a7e3d58
move variables to settings
blink1073 a9de555
add python 3.9 to path
blink1073 b0db9d1
fix path again
blink1073 a3940e3
debug
blink1073 dd59df7
set path
blink1073 3b78e3a
fix path again
blink1073 bcdc2c6
fix up script handling
blink1073 bd1f6c8
fix path
blink1073 23f77e5
use local version of pymongo
blink1073 153cca4
allow podman to work
blink1073 76a248b
remove pull command
blink1073 092a4aa
add debug
blink1073 03fd67f
fix logic
blink1073 3a788a7
remove debug error raise
blink1073 5ec0197
update readme
blink1073 e124409
clean up template
blink1073 090f698
address review
blink1073 a3bb16b
add missing dep
blink1073 0c7a2e2
fix handling of so files
blink1073 b55315b
add missing dep
blink1073 adb246a
fix install
blink1073 809ff6d
use updated path
blink1073 969bb07
use final script versions
blink1073 9178285
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
#!/bin/bash | ||
set -o errexit # Exit the script with error if any of the commands fail | ||
|
||
export PATH="/opt/python/3.9/bin:${PATH}" | ||
python --version | ||
pushd ./test/lambda | ||
|
||
. build.sh | ||
popd | ||
. ${DRIVERS_TOOLS}/.evergreen/run-deployed-lambda-aws-tests.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
AWS Lambda Testing | ||
------------------ | ||
|
||
Running locally | ||
=============== | ||
|
||
Prerequisites: | ||
|
||
- AWS SAM CLI | ||
- Docker daemon running | ||
|
||
Usage | ||
===== | ||
|
||
- Start a local mongodb instance on port 27017 | ||
- Run ``build.sh`` | ||
- Run ``test.sh`` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
#!/bin/bash | ||
set -o errexit # Exit the script with error if any of the commands fail | ||
set -o xtrace | ||
|
||
rm -rf mongodb/pymongo | ||
rm -rf mongodb/gridfs | ||
rm -rf mongodb/bson | ||
|
||
pushd ../.. | ||
rm -f pymongo/*.so | ||
rm -f bson/*.so | ||
image="quay.io/pypa/manylinux2014_x86_64:latest" | ||
|
||
DOCKER=$(command -v docker) || true | ||
if [ -z "$DOCKER" ]; then | ||
PODMAN=$(command -v podman) || true | ||
if [ -z "$PODMAN" ]; then | ||
echo "docker or podman are required!" | ||
exit 1 | ||
fi | ||
DOCKER=podman | ||
fi | ||
|
||
$DOCKER run --rm -v "`pwd`:/src" $image /src/test/lambda/build_internal.sh | ||
cp -r pymongo ./test/lambda/mongodb/pymongo | ||
cp -r bson ./test/lambda/mongodb/bson | ||
cp -r gridfs ./test/lambda/mongodb/gridfs | ||
popd |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
#!/bin/bash -ex | ||
|
||
cd /src | ||
PYTHON=/opt/python/cp39-cp39/bin/python | ||
$PYTHON -m pip install -v -e . |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
{ | ||
"body": "{\"message\": \"hello world\"}", | ||
"resource": "/hello", | ||
"path": "/hello", | ||
"httpMethod": "GET", | ||
"isBase64Encoded": false, | ||
"queryStringParameters": { | ||
"foo": "bar" | ||
}, | ||
"pathParameters": { | ||
"proxy": "/path/to/resource" | ||
}, | ||
"stageVariables": { | ||
"baz": "qux" | ||
}, | ||
"headers": { | ||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", | ||
"Accept-Encoding": "gzip, deflate, sdch", | ||
"Accept-Language": "en-US,en;q=0.8", | ||
"Cache-Control": "max-age=0", | ||
"CloudFront-Forwarded-Proto": "https", | ||
"CloudFront-Is-Desktop-Viewer": "true", | ||
"CloudFront-Is-Mobile-Viewer": "false", | ||
"CloudFront-Is-SmartTV-Viewer": "false", | ||
"CloudFront-Is-Tablet-Viewer": "false", | ||
"CloudFront-Viewer-Country": "US", | ||
"Host": "1234567890.execute-api.us-east-1.amazonaws.com", | ||
"Upgrade-Insecure-Requests": "1", | ||
"User-Agent": "Custom User Agent String", | ||
"Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)", | ||
"X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==", | ||
"X-Forwarded-For": "127.0.0.1, 127.0.0.2", | ||
"X-Forwarded-Port": "443", | ||
"X-Forwarded-Proto": "https" | ||
}, | ||
"requestContext": { | ||
"accountId": "123456789012", | ||
"resourceId": "123456", | ||
"stage": "prod", | ||
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", | ||
"requestTime": "09/Apr/2015:12:34:56 +0000", | ||
"requestTimeEpoch": 1428582896000, | ||
"identity": { | ||
"cognitoIdentityPoolId": null, | ||
"accountId": null, | ||
"cognitoIdentityId": null, | ||
"caller": null, | ||
"accessKey": null, | ||
"sourceIp": "127.0.0.1", | ||
"cognitoAuthenticationType": null, | ||
"cognitoAuthenticationProvider": null, | ||
"userArn": null, | ||
"userAgent": "Custom User Agent String", | ||
"user": null | ||
}, | ||
"path": "/prod/hello", | ||
"resourcePath": "/hello", | ||
"httpMethod": "POST", | ||
"apiId": "1234567890", | ||
"protocol": "HTTP/1.1" | ||
} | ||
} |
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,143 @@ | ||
""" | ||
Lambda function for Python Driver testing | ||
|
||
Creates the client that is cached for all requests, subscribes to | ||
relevant events, and forces the connection pool to get populated. | ||
""" | ||
import json | ||
import os | ||
|
||
from pymongo import MongoClient | ||
from pymongo.monitoring import ( | ||
CommandListener, | ||
ConnectionPoolListener, | ||
ServerHeartbeatListener, | ||
) | ||
|
||
open_connections = 0 | ||
heartbeat_count = 0 | ||
total_heartbeat_duration = 0 | ||
total_commands = 0 | ||
total_command_duration = 0 | ||
|
||
|
||
class CommandHandler(CommandListener): | ||
def started(self, event): | ||
print("command started", event) | ||
|
||
def succeeded(self, event): | ||
global total_commands, total_command_duration | ||
total_commands += 1 | ||
total_command_duration += event.duration_micros / 1e6 | ||
print("command succeeded", event) | ||
|
||
def failed(self, event): | ||
global total_commands, total_command_duration | ||
total_commands += 1 | ||
total_command_duration += event.duration_micros / 1e6 | ||
print("command failed", event) | ||
|
||
|
||
class ServerHeartbeatHandler(ServerHeartbeatListener): | ||
def started(self, event): | ||
print("server heartbeat started", event) | ||
|
||
def succeeded(self, event): | ||
global heartbeat_count, total_heartbeat_duration | ||
heartbeat_count += 1 | ||
total_heartbeat_duration += event.duration | ||
print("server heartbeat succeeded", event) | ||
|
||
def failed(self, event): | ||
global heartbeat_count, total_heartbeat_duration | ||
heartbeat_count += 1 | ||
total_heartbeat_duration += event.duration | ||
print("server heartbeat failed", event) | ||
|
||
|
||
class ConnectionHandler(ConnectionPoolListener): | ||
def connection_created(self, event): | ||
global open_connections | ||
open_connections += 1 | ||
print("connection created") | ||
|
||
def connection_ready(self, event): | ||
pass | ||
|
||
def connection_closed(self, event): | ||
global open_connections | ||
open_connections -= 1 | ||
print("connection closed") | ||
|
||
def connection_check_out_started(self, event): | ||
pass | ||
|
||
def connection_check_out_failed(self, event): | ||
pass | ||
|
||
def connection_checked_out(self, event): | ||
pass | ||
|
||
def connection_checked_in(self, event): | ||
pass | ||
|
||
def pool_created(self, event): | ||
pass | ||
|
||
def pool_ready(self, event): | ||
pass | ||
|
||
def pool_cleared(self, event): | ||
pass | ||
|
||
def pool_closed(self, event): | ||
pass | ||
|
||
|
||
listeners = [CommandHandler(), ServerHeartbeatHandler(), ConnectionHandler()] | ||
print("Creating client") | ||
client = MongoClient(os.environ["MONGODB_URI"], event_listeners=listeners) | ||
|
||
|
||
# Populate the connection pool. | ||
print("Connecting") | ||
client.lambdaTest.list_collections() | ||
print("Connected") | ||
|
||
|
||
# Create the response to send back. | ||
def create_response(): | ||
return dict( | ||
averageCommandDuration=total_command_duration / total_commands, | ||
averageHeartbeatDuration=total_heartbeat_duration / heartbeat_count, | ||
openConnections=open_connections, | ||
heartbeatCount=heartbeat_count, | ||
) | ||
|
||
|
||
# Reset the numbers. | ||
def reset(): | ||
global open_connections, heartbeat_count, total_heartbeat_duration, total_commands, total_command_duration | ||
open_connections = 0 | ||
heartbeat_count = 0 | ||
total_heartbeat_duration = 0 | ||
total_commands = 0 | ||
total_command_duration = 0 | ||
|
||
|
||
def lambda_handler(event, context): | ||
""" | ||
The handler function itself performs an insert/delete and returns the | ||
id of the document in play. | ||
""" | ||
print("initializing") | ||
db = client.lambdaTest | ||
collection = db.test | ||
result = collection.insert_one({"n": 1}) | ||
collection.delete_one({"_id": result.inserted_id}) | ||
# Create the response and then reset the numbers. | ||
response = json.dumps(create_response()) | ||
reset() | ||
print("finished!") | ||
|
||
return dict(statusCode=200, body=response) |
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
#!/bin/bash | ||
set -o errexit # Exit the script with error if any of the commands fail | ||
|
||
sam build | ||
sam local invoke --docker-network host --parameter-overrides "MongoDbUri=mongodb://host.docker.internal:27017" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this need to use rhel87-large? Or can we use rhel87-small instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small works