Description
What happened (please include outputs or screenshots):
Running this simple script in a big cluster takes about 30 seconds to execute:
import kubernetes as k8s
k8s.config.load_kube_config()
apps = k8s.client.AppsV1Api()
print(k8s.__version__)
print(len(apps.list_replica_set_for_all_namespaces().items))
time python3 script:
24.2.0
3661
real 0m32.394s
user 0m17.764s
sys 0m11.235s
as you see it has very high system cpu usage
running this under a profiler shows following interesting things:
957/1 0.002 0.000 32.177 32.177 {built-in method builtins.exec}
1 0.173 0.173 32.177 32.177 test.py:1(<module>)
1 0.000 0.000 31.563 31.563 apps_v1_api.py:3804(list_replica_set_for_all_namespaces)
1 0.000 0.000 31.563 31.563 apps_v1_api.py:3838(list_replica_set_for_all_namespaces_with_http_info)
1 0.000 0.000 31.563 31.563 api_client.py:305(call_api)
1 0.018 0.018 31.563 31.563 api_client.py:120(__call_api)
1 0.000 0.000 28.090 28.090 api_client.py:244(deserialize)
780711/1 0.997 0.000 27.281 27.281 api_client.py:266(__deserialize)
190593/1 0.934 0.000 27.281 27.281 api_client.py:620(__deserialize_model)
36658/1 0.078 0.000 27.281 27.281 api_client.py:280(<listcomp>)
190594 0.839 0.000 22.939 0.000 configuration.py:75(__init__)
190594 0.108 0.000 10.911 0.000 context.py:41(cpu_count)
190594 10.803 0.000 10.803 0.000 {built-in method posix.cpu_count}
190597 0.300 0.000 8.327 0.000 configuration.py:253(debug)
381195 0.216 0.000 7.443 0.000 __init__.py:1448(setLevel)
381195 4.364 0.000 7.117 0.000 __init__.py:1403(_clear_cache)
...
190594 10.803 0.000 10.803 0.000 {built-in method posix.cpu_count}
10 seconds are spent running multiprocessing.cpu_count which accounts for most of the system usage
381195 0.216 0.000 7.443 0.000 __init__.py:1448(setLevel)
7 seconds are spent configuring logging
looking at what causes this appears to be following line in every model:
if local_vars_configuration is None:
local_vars_configuration = Configuration()
This runs the configuration function which sets up logging and calls multiprocessing.cpu_count
commenting the multiprocessing call confirms this, i then runs significantly faster:
real 0m11.964s
user 0m8.162s
sys 0m0.338s
Is there a way to avoid calling Configuration on every model init?