Results of measuring the effectiveness of our startup probe optimization

results.md

Readiness probe optimization results

All of these measurements are taken on my local machine using kind. The usual disclaimers apply. YMMV.

Note: "Adjusted" means HEAD without startup probe on the queue-proxy but HTTP probe on user-container. The following patch is applied to podspeed

diff --git a/pkg/pod/types/knative.go b/pkg/pod/types/knative.go
index 59b2c8d..1aab20b 100644
--- a/pkg/pod/types/knative.go
+++ b/pkg/pod/types/knative.go
@@ -56,6 +56,17 @@ func KnativeHead(ns, name string) *corev1.Pod {
 						},
 					},
 				},
+				ReadinessProbe: &corev1.Probe{
+					Handler: corev1.Handler{
+						HTTPGet: &corev1.HTTPGetAction{
+							Port: intstr.FromInt(8080),
+						},
+					},
+					TimeoutSeconds:   1,
+					FailureThreshold: 3,
+					SuccessThreshold: 1,
+					PeriodSeconds:    1,
+				},
 				TerminationMessagePolicy: corev1.TerminationMessageFallbackToLogsOnError,
 			}, {
 				Name:            "queue-proxy",
@@ -175,17 +186,6 @@ func KnativeHead(ns, name string) *corev1.Pod {
 						corev1.ResourceCPU: resource.MustParse("25m"),
 					},
 				},
-				StartupProbe: &corev1.Probe{
-					Handler: corev1.Handler{
-						Exec: &corev1.ExecAction{
-							Command: []string{"/ko-app/queue", "-probe-timeout", "10m0s"},
-						},
-					},
-					TimeoutSeconds:   600,
-					FailureThreshold: 1,
-					SuccessThreshold: 1,
-					PeriodSeconds:    1,
-				},
 				ReadinessProbe: &corev1.Probe{
 					Handler: corev1.Handler{
 						HTTPGet: &corev1.HTTPGetAction{

Note: "Rolled back" means HEAD but reverting the qp's startup probe to a readiness probe. The following patch was applied

diff --git a/pkg/pod/types/knative.go b/pkg/pod/types/knative.go
index 59b2c8d..c184698 100644
--- a/pkg/pod/types/knative.go
+++ b/pkg/pod/types/knative.go
@@ -175,28 +175,13 @@ func KnativeHead(ns, name string) *corev1.Pod {
 						corev1.ResourceCPU: resource.MustParse("25m"),
 					},
 				},
-				StartupProbe: &corev1.Probe{
+				ReadinessProbe: &corev1.Probe{
 					Handler: corev1.Handler{
 						Exec: &corev1.ExecAction{
 							Command: []string{"/ko-app/queue", "-probe-timeout", "10m0s"},
 						},
 					},
 					TimeoutSeconds:   600,
-					FailureThreshold: 1,
-					SuccessThreshold: 1,
-					PeriodSeconds:    1,
-				},
-				ReadinessProbe: &corev1.Probe{
-					Handler: corev1.Handler{
-						HTTPGet: &corev1.HTTPGetAction{
-							HTTPHeaders: []corev1.HTTPHeader{{
-								Name:  "K-Network-Probe",
-								Value: "queue",
-							}},
-							Port: intstr.FromInt(8012),
-						},
-					},
-					TimeoutSeconds:   1,
 					FailureThreshold: 3,
 					SuccessThreshold: 1,
 					PeriodSeconds:    1,

Kubernetes 1.22.0

HEAD

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |2    |13   |5    |8    |10
Time to ip        |686  |1988 |1218 |1813 |1900
Time to ready     |1695 |2997 |2292 |2985 |2991

Adjusted

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |0    |7    |5    |7    |7
Time to ip        |1175 |1785 |1412 |1593 |1689
Time to ready     |1833 |2594 |2129 |2587 |2590

Rolled back

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |0    |9    |4    |7    |8
Time to ip        |966  |1964 |1430 |1958 |1961
Time to ready     |1006 |2190 |1788 |2059 |2124

Kubernetes 1.21.1

HEAD

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |0    |7    |4    |7    |7
Time to ip        |844  |1424 |1116 |1393 |1408
Time to ready     |2013 |3029 |2619 |3026 |3028

Adjusted

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |0    |9    |4    |8    |8
Time to ip        |754  |1832 |1366 |1813 |1822
Time to ready     |1007 |2021 |1828 |2018 |2020

Rolled back

Created 20 knative-head pods sequentially, results are in ms:

metric            |min |max  |mean |p95  |p99
Time to scheduled |2   |9    |4    |5    |7
Time to ip        |719 |1963 |1310 |1803 |1883
Time to ready     |751 |2001 |1498 |1842 |1922

Just readiness probe

Created 20 knative-head pods sequentially, results are in ms:

metric            |min |max  |mean |p95  |p99
Time to scheduled |2   |12   |4    |7    |10
Time to ip        |792 |1863 |1326 |1843 |1853
Time to ready     |805 |1878 |1498 |1852 |1865

Kubernetes 1.20.7

HEAD

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |2    |11   |4    |6    |8
Time to ip        |619  |1566 |1091 |1560 |1563
Time to ready     |1056 |3300 |2080 |3060 |3180

Adjusted

Created 20 knative-head pods sequentially, results are in ms:

metric            |min  |max  |mean |p95  |p99
Time to scheduled |0    |5    |3    |5    |5
Time to ip        |822  |1482 |1211 |1434 |1458
Time to ready     |1494 |2211 |1912 |2188 |2200

Rolled back

Created 20 knative-head pods sequentially, results are in ms:

metric            |min |max  |mean |p95  |p99
Time to scheduled |2   |11   |4    |5    |8
Time to ip        |539 |1657 |1102 |1525 |1591
Time to ready     |806 |2505 |1503 |1881 |2193