Skip to content

PYTHON-3461 Test FaaS (AWS Lambda) Behavior Per Driver #1310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Aug 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
525e29b
PYTHON-3461 Test FaaS (AWS Lambda) Behavior Per Driver
blink1073 Jul 14, 2023
0e82ac1
lint
blink1073 Jul 14, 2023
fae67e2
add test and EG config
blink1073 Jul 14, 2023
8220e3b
clean up
blink1073 Jul 16, 2023
a35f4e5
get the function working locally
blink1073 Jul 17, 2023
8d0e455
add missing setup
blink1073 Jul 17, 2023
a7e3d58
move variables to settings
blink1073 Jul 17, 2023
a9de555
add python 3.9 to path
blink1073 Jul 18, 2023
b0db9d1
fix path again
blink1073 Jul 18, 2023
a3940e3
debug
blink1073 Jul 18, 2023
dd59df7
set path
blink1073 Jul 18, 2023
3b78e3a
fix path again
blink1073 Jul 18, 2023
bcdc2c6
fix up script handling
blink1073 Jul 18, 2023
bd1f6c8
fix path
blink1073 Jul 18, 2023
23f77e5
use local version of pymongo
blink1073 Jul 19, 2023
153cca4
allow podman to work
blink1073 Jul 19, 2023
76a248b
remove pull command
blink1073 Jul 19, 2023
092a4aa
add debug
blink1073 Jul 19, 2023
03fd67f
fix logic
blink1073 Jul 19, 2023
3a788a7
remove debug error raise
blink1073 Jul 19, 2023
5ec0197
update readme
blink1073 Jul 19, 2023
e124409
clean up template
blink1073 Jul 19, 2023
090f698
address review
blink1073 Jul 25, 2023
a3bb16b
add missing dep
blink1073 Jul 25, 2023
0c7a2e2
fix handling of so files
blink1073 Jul 26, 2023
b55315b
add missing dep
blink1073 Jul 26, 2023
adb246a
fix install
blink1073 Jul 26, 2023
809ff6d
use updated path
blink1073 Aug 3, 2023
969bb07
use final script versions
blink1073 Aug 11, 2023
9178285
Merge branch 'master' of github.com:mongodb/mongo-python-driver into …
blink1073 Aug 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .evergreen/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1303,6 +1303,33 @@ task_groups:
tasks:
- testazurekms-task

- name: test_aws_lambda_task_group
setup_group:
- func: fetch source
- func: prepare resources
- command: subprocess.exec
params:
working_dir: src
binary: bash
add_expansions_to_env: true
args:
- ${DRIVERS_TOOLS}/.evergreen/atlas/setup-atlas-cluster.sh
- command: expansions.update
params:
file: src/atlas-expansion.yml
teardown_group:
- command: subprocess.exec
params:
working_dir: src
binary: bash
add_expansions_to_env: true
args:
- ${DRIVERS_TOOLS}/.evergreen/atlas/teardown-atlas-cluster.sh
setup_group_can_fail_task: true
setup_group_timeout_secs: 1800
tasks:
- test-aws-lambda-deployed

- name: test_atlas_task_group_search_indexes
setup_group:
- func: fetch source
Expand Down Expand Up @@ -1785,6 +1812,23 @@ tasks:
vars:
TEST_DATA_LAKE: "true"

- name: "test-aws-lambda-deployed"
commands:
- func: "install dependencies"
- command: ec2.assume_role
params:
role_arn: ${LAMBDA_AWS_ROLE_ARN}
duration_seconds: 3600
- command: subprocess.exec
params:
working_dir: src
binary: bash
add_expansions_to_env: true
args:
- .evergreen/run-deployed-lambda-aws-tests.sh
env:
TEST_LAMBDA_DIRECTORY: ${PROJECT_DIRECTORY}/test/lambda

- name: test-ocsp-rsa-valid-cert-server-staples
tags: ["ocsp", "ocsp-rsa", "ocsp-staple"]
commands:
Expand Down Expand Up @@ -3358,6 +3402,12 @@ buildvariants:
batchtime: 20160 # Use a batchtime of 14 days as suggested by the CSFLE test README
- testazurekms-fail-task

- name: rhel8-test-lambda
display_name: AWS Lambda handler tests
run_on: rhel87-small
tasks:
- name: test_aws_lambda_task_group

- name: Release
display_name: Release
batchtime: 20160 # 14 days
Expand Down
10 changes: 10 additions & 0 deletions .evergreen/run-deployed-lambda-aws-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash
set -o errexit # Exit the script with error if any of the commands fail

export PATH="/opt/python/3.9/bin:${PATH}"
python --version
pushd ./test/lambda

. build.sh
popd
. ${DRIVERS_TOOLS}/.evergreen/aws_lambda/run-deployed-lambda-aws-tests.sh
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,10 @@ mongocryptd.pid
.idea/
.nova/
venv/

# Lambda temp files
test/lambda/.aws-sam
test/lambda/env.json
test/lambda/mongodb/pymongo/*
test/lambda/mongodb/gridfs/*
test/lambda/mongodb/bson/*
1 change: 1 addition & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ repos:
- id: check-case-conflict
- id: check-toml
- id: check-yaml
exclude: template.yaml
- id: debug-statements
- id: end-of-file-fixer
exclude: WHEEL
Expand Down
17 changes: 17 additions & 0 deletions test/lambda/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
AWS Lambda Testing
------------------

Running locally
===============

Prerequisites:

- AWS SAM CLI
- Docker daemon running

Usage
=====

- Start a local mongodb instance on port 27017
- Run ``build.sh``
- Run ``test.sh``
28 changes: 28 additions & 0 deletions test/lambda/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash
set -o errexit # Exit the script with error if any of the commands fail
set -o xtrace

rm -rf mongodb/pymongo
rm -rf mongodb/gridfs
rm -rf mongodb/bson

pushd ../..
rm -f pymongo/*.so
rm -f bson/*.so
image="quay.io/pypa/manylinux2014_x86_64:latest"

DOCKER=$(command -v docker) || true
if [ -z "$DOCKER" ]; then
PODMAN=$(command -v podman) || true
if [ -z "$PODMAN" ]; then
echo "docker or podman are required!"
exit 1
fi
DOCKER=podman
fi

$DOCKER run --rm -v "`pwd`:/src" $image /src/test/lambda/build_internal.sh
cp -r pymongo ./test/lambda/mongodb/pymongo
cp -r bson ./test/lambda/mongodb/bson
cp -r gridfs ./test/lambda/mongodb/gridfs
popd
5 changes: 5 additions & 0 deletions test/lambda/build_internal.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash -ex

cd /src
PYTHON=/opt/python/cp39-cp39/bin/python
$PYTHON -m pip install -v -e .
62 changes: 62 additions & 0 deletions test/lambda/events/event.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"body": "{\"message\": \"hello world\"}",
"resource": "/hello",
"path": "/hello",
"httpMethod": "GET",
"isBase64Encoded": false,
"queryStringParameters": {
"foo": "bar"
},
"pathParameters": {
"proxy": "/path/to/resource"
},
"stageVariables": {
"baz": "qux"
},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, sdch",
"Accept-Language": "en-US,en;q=0.8",
"Cache-Control": "max-age=0",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Host": "1234567890.execute-api.us-east-1.amazonaws.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Custom User Agent String",
"Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
"X-Forwarded-For": "127.0.0.1, 127.0.0.2",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"requestContext": {
"accountId": "123456789012",
"resourceId": "123456",
"stage": "prod",
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
"requestTime": "09/Apr/2015:12:34:56 +0000",
"requestTimeEpoch": 1428582896000,
"identity": {
"cognitoIdentityPoolId": null,
"accountId": null,
"cognitoIdentityId": null,
"caller": null,
"accessKey": null,
"sourceIp": "127.0.0.1",
"cognitoAuthenticationType": null,
"cognitoAuthenticationProvider": null,
"userArn": null,
"userAgent": "Custom User Agent String",
"user": null
},
"path": "/prod/hello",
"resourcePath": "/hello",
"httpMethod": "POST",
"apiId": "1234567890",
"protocol": "HTTP/1.1"
}
}
4 changes: 4 additions & 0 deletions test/lambda/mongodb/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

build-MongoDBFunction:
cp -r . $(ARTIFACTS_DIR)
python -m pip install -t $(ARTIFACTS_DIR) dnspython
Empty file added test/lambda/mongodb/__init__.py
Empty file.
149 changes: 149 additions & 0 deletions test/lambda/mongodb/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
"""
Lambda function for Python Driver testing

Creates the client that is cached for all requests, subscribes to
relevant events, and forces the connection pool to get populated.
"""
import json
import os

from bson import has_c as has_bson_c
from pymongo import MongoClient
from pymongo import has_c as has_pymongo_c
from pymongo.monitoring import (
CommandListener,
ConnectionPoolListener,
ServerHeartbeatListener,
)

open_connections = 0
heartbeat_count = 0
total_heartbeat_duration = 0
total_commands = 0
total_command_duration = 0

# Ensure we are using C extensions
assert has_bson_c()
assert has_pymongo_c()


class CommandHandler(CommandListener):
def started(self, event):
print("command started", event)

def succeeded(self, event):
global total_commands, total_command_duration
total_commands += 1
total_command_duration += event.duration_micros / 1e6
print("command succeeded", event)

def failed(self, event):
global total_commands, total_command_duration
total_commands += 1
total_command_duration += event.duration_micros / 1e6
print("command failed", event)


class ServerHeartbeatHandler(ServerHeartbeatListener):
def started(self, event):
print("server heartbeat started", event)

def succeeded(self, event):
global heartbeat_count, total_heartbeat_duration
heartbeat_count += 1
total_heartbeat_duration += event.duration
print("server heartbeat succeeded", event)

def failed(self, event):
global heartbeat_count, total_heartbeat_duration
heartbeat_count += 1
total_heartbeat_duration += event.duration
print("server heartbeat failed", event)


class ConnectionHandler(ConnectionPoolListener):
def connection_created(self, event):
global open_connections
open_connections += 1
print("connection created")

def connection_ready(self, event):
pass

def connection_closed(self, event):
global open_connections
open_connections -= 1
print("connection closed")

def connection_check_out_started(self, event):
pass

def connection_check_out_failed(self, event):
pass

def connection_checked_out(self, event):
pass

def connection_checked_in(self, event):
pass

def pool_created(self, event):
pass

def pool_ready(self, event):
pass

def pool_cleared(self, event):
pass

def pool_closed(self, event):
pass


listeners = [CommandHandler(), ServerHeartbeatHandler(), ConnectionHandler()]
print("Creating client")
client = MongoClient(os.environ["MONGODB_URI"], event_listeners=listeners)


# Populate the connection pool.
print("Connecting")
client.lambdaTest.list_collections()
print("Connected")


# Create the response to send back.
def create_response():
return dict(
averageCommandDuration=total_command_duration / total_commands,
averageHeartbeatDuration=total_heartbeat_duration / heartbeat_count,
openConnections=open_connections,
heartbeatCount=heartbeat_count,
)


# Reset the numbers.
def reset():
global open_connections, heartbeat_count, total_heartbeat_duration, total_commands, total_command_duration
open_connections = 0
heartbeat_count = 0
total_heartbeat_duration = 0
total_commands = 0
total_command_duration = 0


def lambda_handler(event, context):
"""
The handler function itself performs an insert/delete and returns the
id of the document in play.
"""
print("initializing")
db = client.lambdaTest
collection = db.test
result = collection.insert_one({"n": 1})
collection.delete_one({"_id": result.inserted_id})
# Create the response and then reset the numbers.
response = json.dumps(create_response())
reset()
print("finished!")

return dict(statusCode=200, body=response)
5 changes: 5 additions & 0 deletions test/lambda/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash
set -o errexit # Exit the script with error if any of the commands fail

sam build
sam local invoke --docker-network host --parameter-overrides "MongoDbUri=mongodb://host.docker.internal:27017"
Loading