Kubernetes client exception with connection refused error

[hardys666]

Our tests had been running fine till we started running into the following issue where the kubernetes client websocket call will terminate with an exception:

channel = stream(api.connect_get_namespaced_pod_exec,

/usr/lib/python3.8/site-packages/kubernetes/stream/stream.py:35: in _websocket_request
return api_method(*args, **kwargs)
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:994: in connect_get_namespaced_pod_exec
return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501
/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:1101: in connect_get_namespaced_pod_exec_with_http_info
return self.api_client.call_api(
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
return self.__call_api(resource_path, method,
/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
response_data = self.request(

---

configuration = <kubernetes.client.configuration.Configuration object at 0x7f192844cb20>
_method = 'GET'
url = 'wss://api.ndal-ocp2.ve.pd.f5net.com:6443/api/v1/namespaces/spk-test-automation/pods/ans-client-ci6492742-127/exec?com...command=2001%3A0db8%3A0%3A127%3A%3A100&command=-t&command=10&command=-P8&stderr=True&stdin=False&stdout=True&tty=False'
kwargs = {'_preload_content': False, '_request_timeout': None, 'body': None, 'headers': {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}, ...}
headers = {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}
_request_timeout = None, _preload_content = False, capture_all = True
def websocket_call(configuration, _method, url, **kwargs):
"""An internal function to be called in api-client when a websocket
connection is required. method, url, and kwargs are the parameters of
apiClient.request method."""

    url = get_websocket_url(url, kwargs.get("query_params"))
    headers = kwargs.get("headers")
    _request_timeout = kwargs.get("_request_timeout", 60)
    _preload_content = kwargs.get("_preload_content", True)
    capture_all = kwargs.get("capture_all", True)

    try:
        client = WSClient(configuration, url, headers, capture_all)
        if not _preload_content:
            return client
        client.run_forever(timeout=_request_timeout)
        return WSResponse('%s' % ''.join(client.read_all()))
    except (Exception, KeyboardInterrupt, SystemExit) as e:

      raise ApiException(status=0, reason=str(e))

E kubernetes.client.exceptions.ApiException: (0)
E Reason: [Errno 111] Connection refused
/usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py:525: ApiException

These exceptions happen at random (not for any specific or every test) and across different kubernetes environment/clusters. They started happening somewhere around early Feb, 2023. (not sure if its related to any fixes/changes).

Below is what we use in our code:
`def k8s_exec(command, pod, namespace, api):
channel = stream(api.connect_get_namespaced_pod_exec,
pod,
namespace,
command=command,
stderr=True, stdin=False,
stdout=True, tty=False,
_preload_content=False)
channel.run_forever(timeout=60)
err = yaml.safe_load(channel.read_channel(ERROR_CHANNEL))
if err is None:
raise AssertionError(f'The command "{command}" was timeouted')
rc = 0 if err['status'] == 'Success' else 1
stdout = channel.read_stdout()
stderr = channel.read_stderr()

return rc, stdout, stderr`

Not sure if this is similar to: 1841

What you expected to happen:
Connection is retained.

Environment:

• Kubernetes version (kubectl version):
  Client Version: version.Info{Major:"1", Minor:"20"
  Server Version: version.Info{Major:"1", Minor:"23"
• OS (e.g., MacOS 10.13.6): Linux
• Python version (python --version): 2.7.5