|
| 1 | +.. _sky-api-server-performance-best-practices: |
| 2 | + |
| 3 | +SkyPilot API Server Performance Best Practices |
| 4 | +============================================== |
| 5 | + |
| 6 | +This page describes performance best practices for remote SkyPilot API server in :ref:`team deployment <sky-api-server>`. |
| 7 | + |
| 8 | +Tuning API server resources |
| 9 | +--------------------------- |
| 10 | + |
| 11 | +The number of requests that the API server can handle concurrently is proportional to the resources (CPU cores and memory) allocated to it. Internally, requests are categorized into two different types and handled in a first-in-first-out manner: |
| 12 | + |
| 13 | +* ``Long-running requests``: requests that take longer time and more resources to run, including ``launch``, ``exec``, ``jobs.launch``, etc. |
| 14 | +* ``Short-running requests``: requests that take shorter time or less resources to run, including ``status``, ``logs``, etc. |
| 15 | + |
| 16 | +.. note:: |
| 17 | + |
| 18 | + Requests are queued and processed by the API server. Therefore, they only take resources off the API server when they are in queue or being processed. Once requests are processed and remote clusters start doing real work, they no longer require API server's resources or count against its concurrency limit. |
| 19 | + |
| 20 | + For example, long-running requests for ``launch`` and ``exec`` no longer take resources off the API server once a cluster has been provisioned, or once a job has been submitted to a cluster, respectively. |
| 21 | + |
| 22 | +It is recommended to tune the resources allocated to the API server based on the expected concurrency to avoid queuing: |
| 23 | + |
| 24 | +.. tab-set:: |
| 25 | + |
| 26 | + .. tab-item:: Helm Deployment |
| 27 | + |
| 28 | + .. code-block:: bash |
| 29 | +
|
| 30 | + # NAMESPACE and RELEASE_NAME should be the same as the deployment step |
| 31 | + helm upgrade -n ${NAMESPACE} ${RELEASE_NAME} skypilot/skypilot-nightly \ |
| 32 | + --reuse-values \ |
| 33 | + --set apiService.resources.requests.cpu=8 \ |
| 34 | + --set apiService.resources.requests.memory=16Gi \ |
| 35 | + --set apiService.resources.limits.cpu=8 \ |
| 36 | + --set apiService.resources.limits.memory=16Gi |
| 37 | + |
| 38 | + .. note:: |
| 39 | + |
| 40 | + If you specify a resources that is lower than the recommended minimum resources for team usage (4 CPUs with 8GB of memory, which is also the default value when ``apiService.resources`` are not specified), an error will be raised on ``helm upgrade``. You can specify ``--set apiService.skipResourcesCheck=true`` to skip the check if performance and stability is not an issue for you scenario. |
| 41 | + |
| 42 | + .. dropdown:: Why set the same value for the limits and requests? |
| 43 | + |
| 44 | + We recommend setting the resources limits to the same value as the requests for the following reasons: |
| 45 | + |
| 46 | + * API server depends on the resources limit to detect the available resources and decide the maximum concurrency. Setting limits larger than the requests or omitting the limits will cause the API server make aggressive concurrency decisions and may cause high resource contention on the Kubernetes node. |
| 47 | + * Setting the same value for the limits and requests ensures the QoS class of the API server pod being set to ``Guaranteed`` and reduce the chance of the pod being killed by the Kubernetes node when the node is under resource pressure. |
| 48 | + |
| 49 | + In short, setting the same value for the limits and requests sacrifices the resources utilization for stability and predictability. Pivoting to other trade-off is also possible, but we recommend to keep the memory request and limit the same in production environment to avoid potential eviction caused by node memory pressure. |
| 50 | + |
| 51 | + .. tab-item:: VM Deployment |
| 52 | + |
| 53 | + For VM deployment, in-place vertically scaling the API server is not supported and the API server need to be terminated and recreated to apply the new resources. This means the current state of the API server will be lost. We recommend to create an new API server instance with the new resources and gradually migrate the workload to the new API server. |
| 54 | + |
| 55 | + Refer to :ref:`sky-api-server-cloud-deploy` for how to deploy the new API server and modify the cluster configuration before running ``sky launch``: |
| 56 | + |
| 57 | + .. code-block:: diff |
| 58 | +
|
| 59 | + resources: |
| 60 | + - cpus: 8+ |
| 61 | + - memory: 16+ |
| 62 | + + cpus: 16+ |
| 63 | + + memory: 32+ |
| 64 | +
|
| 65 | +The following table shows the maximum concurrency for different resource configurations: |
| 66 | + |
| 67 | +.. list-table:: |
| 68 | + :widths: 1 1 3 2 |
| 69 | + :header-rows: 1 |
| 70 | + |
| 71 | + * - CPU |
| 72 | + - Memory |
| 73 | + - Max concurrency |
| 74 | + - Suggested team size |
| 75 | + * - 4 |
| 76 | + - 8Gi |
| 77 | + - 8 Long requests, 11 Short requests |
| 78 | + - 1~5 users |
| 79 | + * - 16 |
| 80 | + - 32Gi |
| 81 | + - 32 Long requests, 60 Short requests |
| 82 | + - 1~15 users |
| 83 | + * - 32 |
| 84 | + - 64Gi |
| 85 | + - 64 Long requests, 145 Short requests |
| 86 | + - 1~30 users |
| 87 | + * - 64 |
| 88 | + - 128Gi |
| 89 | + - 128 Long requests, 299 Short requests |
| 90 | + - 10~50 users |
| 91 | + * - 128 |
| 92 | + - 256Gi |
| 93 | + - 256 Long requests, 589 Short requests |
| 94 | + - 10~100 users |
| 95 | + |
| 96 | +Use asynchronous requests to avoid blocking |
| 97 | +------------------------------------------- |
| 98 | + |
| 99 | +There is no limit on the number of queued requests. To avoid request blocking, you can either (1) allocate more resources to increase the maximum concurrency (described above), or (2) :ref:`submit requests asynchronously <async>` (``--async``) and poll the status asynchronously. |
| 100 | + |
| 101 | +For example: |
| 102 | + |
| 103 | +.. code-block:: bash |
| 104 | +
|
| 105 | + # Submit 2000 jobs at once without blocking |
| 106 | + for i in `seq 1 2000`; do |
| 107 | + sky jobs launch -y --async job.yaml |
| 108 | + done |
| 109 | + # Poll the status of the jobs |
| 110 | + watch -n 5 "sky jobs queue" |
| 111 | +
|
| 112 | +The requests will be queued on the API server and be processed in submission order. If you find the status is not updated for a while, you can inspect the status of the submitted requests with: |
| 113 | + |
| 114 | +.. code-block:: console |
| 115 | +
|
| 116 | + $ sky api status |
| 117 | + ID User Name Created Status |
| 118 | + d9c95c7e-d248-4a7f-b72e-636511405357 alice sky.jobs.launch a few secs ago PENDING |
| 119 | + 767182fd-0202-4ae5-b2d7-ddfabea5c821 alice sky.jobs.launch a few secs ago PENDING |
| 120 | + 5667cff2-e953-4b80-9e5f-546cea83dc59 alice sky.jobs.launch a few secs ago RUNNING |
| 121 | +
|
| 122 | +Checking the logs of a request |
| 123 | +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 124 | + |
| 125 | +There should be some ``RUNNING`` requests that occupy the concurrency limit. Usually the ``RUNNING`` requests make progress and finally your requests will be processed, but if the ``RUNNING`` requests are stuck, you can inspect the request log with: |
| 126 | + |
| 127 | +.. code-block:: console |
| 128 | +
|
| 129 | + # Replace <request_id> with the actual request id from the ID column |
| 130 | + $ sky api logs <request_id> |
| 131 | +
|
| 132 | +Canceling a request |
| 133 | +^^^^^^^^^^^^^^^^^^^ |
| 134 | + |
| 135 | +If the request is stuck according to the log, e.g. retrying to launch VMs that is out of stock, you can cancel the request with: |
| 136 | + |
| 137 | +.. code-block:: bash |
| 138 | +
|
| 139 | + sky api cancel <requst_id> |
| 140 | +
|
| 141 | +Avoid concurrent logs requests |
| 142 | +------------------------------ |
| 143 | + |
| 144 | +If you run ``sky logs`` to tail the logs of a task, the log tailing will keep taking resources off the API server as long as the task being tailed is still running. Thus, concurrent log requests will occupy the concurrency limit and potentially delay other requests. |
| 145 | + |
| 146 | +To avoid this, it is recommended to run ``sky logs`` and ``sky jobs logs`` with ``--no-follow`` flag if there is a large number of concurrent log requests: |
| 147 | + |
| 148 | +.. code-block:: bash |
| 149 | +
|
| 150 | + sky logs --no-follow my_cluster |
| 151 | +
|
| 152 | +Commands that execute tasks like ``sky jobs launch`` and ``sky exec`` will also tail the logs of the task after the task is started by default. You can add ``--async`` flag to submit the job without tailing the logs: |
| 153 | + |
| 154 | +.. code-block:: bash |
| 155 | +
|
| 156 | + sky jobs launch --async job.yaml |
0 commit comments