@int128

int128.hatenablog.com

Upgrade to Kubernetes 1.9 using kops

個人用のKubernetesクラスタを1.9.3に更新したのでログを残します。

% kops version
Version 1.9.0

kops upgrade cluster コマンドでS3の構成情報を更新します。

% kops upgrade cluster
Using cluster from kubectl context: hello.k8s.local

ITEM    PROPERTY        OLD NEW
Cluster KubernetesVersion   1.8.7   1.9.3

Must specify --yes to perform upgrade

% kops upgrade cluster --yes
Using cluster from kubectl context: hello.k8s.local

ITEM    PROPERTY        OLD NEW
Cluster KubernetesVersion   1.8.7   1.9.3

Updates applied to configuration.
You can now apply these changes, using `kops update cluster hello.k8s.local`

kops update cluster コマンドでAWSリソースの設定(Launch Config等)を更新します。

% kops update cluster
Using cluster from kubectl context: hello.k8s.local

I0420 20:07:38.893483   47153 executor.go:91] Tasks: 0 done / 77 total; 31 can run
I0420 20:07:42.825316   47153 executor.go:91] Tasks: 31 done / 77 total; 26 can run
I0420 20:07:45.897275   47153 executor.go:91] Tasks: 57 done / 77 total; 18 can run
I0420 20:07:48.800846   47153 executor.go:91] Tasks: 75 done / 77 total; 2 can run
I0420 20:07:49.113445   47153 executor.go:91] Tasks: 77 done / 77 total; 0 can run
Will modify resources:
  DHCPOptions/hello.k8s.local
    Tags                     {KubernetesCluster: hello.k8s.local, Name: hello.k8s.local} -> {Name: hello.k8s.local, KubernetesCluster: hello.k8s.local, kubernetes.io/cluster/hello.k8s.local: owned}

  EBSVolume/a.etcd-events.hello.k8s.local
    VolumeType               standard -> gp2
    Tags                     {k8s.io/etcd/events: a/a, KubernetesCluster: hello.k8s.local, Name: a.etcd-events.hello.k8s.local, k8s.io/role/master: 1} -> {k8s.io/etcd/events: a/a, k8s.io/role/master: 1, kubernetes.io/cluster/hello.k8s.local: owned, Name: a.etcd-events.hello.k8s.local, KubernetesCluster: hello.k8s.local}

  EBSVolume/a.etcd-main.hello.k8s.local
    VolumeType               standard -> gp2
    Tags                     {k8s.io/role/master: 1, k8s.io/etcd/main: a/a, Name: a.etcd-main.hello.k8s.local, KubernetesCluster: hello.k8s.local} -> {k8s.io/etcd/main: a/a, k8s.io/role/master: 1, kubernetes.io/cluster/hello.k8s.local: owned, Name: a.etcd-main.hello.k8s.local, KubernetesCluster: hello.k8s.local}

  InternetGateway/hello.k8s.local
    Tags                     {KubernetesCluster: hello.k8s.local, Name: hello.k8s.local} -> {Name: hello.k8s.local, KubernetesCluster: hello.k8s.local, kubernetes.io/cluster/hello.k8s.local: owned}

  LaunchConfiguration/master-us-west-2a.masters.hello.k8s.local
    UserData
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.9.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.8.1/linux/amd64/nodeup
                            + NODEUP_HASH=54ecae66a2b4e1409b36fc00b550f2501afedbfc
                            +
                            - NODEUP_HASH=bb41724c37d15ab7e039e06230e742b9b38d0808


                            ...
                                - max-file=5
                                storage: overlay,aufs
                            +   version: 17.03.2
                            -   version: 1.13.1
                              encryptionConfig: null
                              etcdClusters:
                                events:
                            -     version: 3.2.14
                            +     image: gcr.io/google_containers/etcd:3.2.14
                            +     version: 3.2.14
                            +   main:
                            +     image: gcr.io/google_containers/etcd:3.2.14
                            -   main:
                                  version: 3.2.14
                              kubeAPIServer:
                            ...
                                - DefaultStorageClass
                                - DefaultTolerationSeconds
                            +   - MutatingAdmissionWebhook
                            +   - ValidatingAdmissionWebhook
                                - NodeRestriction
                                - ResourceQuota
                            ...
                                etcdServersOverrides:
                                - /events#http://127.0.0.1:4002
                            +   image: gcr.io/google_containers/kube-apiserver:v1.9.3
                            -   image: k8s.gcr.io/kube-apiserver:v1.8.7
                                insecurePort: 8080
                                kubeletPreferredAddressTypes:
                            ...
                                clusterName: hello.k8s.local
                                configureCloudRoutes: true
                            +   image: gcr.io/google_containers/kube-controller-manager:v1.9.3
                            -   image: k8s.gcr.io/kube-controller-manager:v1.8.7
                                leaderElection:
                                  leaderElect: true
                            ...
                                clusterCIDR: 100.96.0.0/11
                                cpuRequest: 100m
                            -   featureGates: null
                                hostnameOverride: '@aws'
                            +   image: gcr.io/google_containers/kube-proxy:v1.9.3
                            -   image: k8s.gcr.io/kube-proxy:v1.8.7
                                logLevel: 2
                              kubeScheduler:
                            +   image: gcr.io/google_containers/kube-scheduler:v1.9.3
                            -   image: k8s.gcr.io/kube-scheduler:v1.8.7
                                leaderElection:
                                  leaderElect: true
                            ...
                                networkPluginName: kubenet
                                nonMasqueradeCIDR: 100.64.0.0/10
                            +   podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
                            -   podInfraContainerImage: k8s.gcr.io/pause-amd64:3.0
                                podManifestPath: /etc/kubernetes/manifests
                            -   requireKubeconfig: true
                              masterKubelet:
                                allowPrivileged: true
                            ...
                                networkPluginName: kubenet
                                nonMasqueradeCIDR: 100.64.0.0/10
                            +   podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
                            -   podInfraContainerImage: k8s.gcr.io/pause-amd64:3.0
                                podManifestPath: /etc/kubernetes/manifests
                                registerSchedulable: false
                            -   requireKubeconfig: true

                              __EOF_CLUSTER_SPEC
                            ...
                              nodeLabels:
                                kops.k8s.io/instancegroup: master-us-west-2a
                            + suspendProcesses: null
                              taints: null

                            ...
                              cat > kube_env.yaml << '__EOF_KUBE_ENV'
                              Assets:
                            + - ef979a00ba2f7bf4ee5023e82f94ced2d94c1726@https://storage.googleapis.com/kubernetes-release/release/v1.9.3/bin/linux/amd64/kubelet
                            - - 0f3a59e4c0aae8c2b2a0924d8ace010ebf39f48e@https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubelet
                            + - a27d808eb011dbeea876fe5326349ed167a7ed28@https://storage.googleapis.com/kubernetes-release/release/v1.9.3/bin/linux/amd64/kubectl
                            - - 36340bb4bb158357fe36ffd545d8295774f55ed9@https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
                            - - 1d9788b0f5420e1a219aad2cb8681823fc515e7c@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
                            + - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - c6f310214f687b6c2f32e81c2a49235182950be3@https://kubeupv2.s3.amazonaws.com/kops/1.9.0/linux/amd64/utils.tar.gz
                            - - 42b15a0a0a56531750bde3c7b08d0cf27c170c48@https://kubeupv2.s3.amazonaws.com/kops/1.8.1/linux/amd64/utils.tar.gz
                              ClusterName: hello.k8s.local
                              ConfigBase: s3://state.hello.k8s.local/hello.k8s.local
                            ...
                              - s3://state.hello.k8s.local/hello.k8s.local/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 4bbfcc6df1c1c0953bd0532113a74b7ae21e0ded
                            -   hash: 0b1f26208f8f6cc02468368706d0236670fec8a2
                            +   name: protokube:1.9.0
                            -   name: protokube:1.8.1
                            +   source: https://kubeupv2.s3.amazonaws.com/kops/1.9.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.8.1/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...


  LaunchConfiguration/nodes.hello.k8s.local
    UserData
                            ...
                              set -o pipefail

                            + NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.9.0/linux/amd64/nodeup
                            - NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.8.1/linux/amd64/nodeup
                            + NODEUP_HASH=54ecae66a2b4e1409b36fc00b550f2501afedbfc
                            +
                            - NODEUP_HASH=bb41724c37d15ab7e039e06230e742b9b38d0808


                            ...
                                - max-file=5
                                storage: overlay,aufs
                            +   version: 17.03.2
                            -   version: 1.13.1
                              kubeProxy:
                                clusterCIDR: 100.96.0.0/11
                                cpuRequest: 100m
                            -   featureGates: null
                                hostnameOverride: '@aws'
                            +   image: gcr.io/google_containers/kube-proxy:v1.9.3
                            -   image: k8s.gcr.io/kube-proxy:v1.8.7
                                logLevel: 2
                              kubelet:
                            ...
                                networkPluginName: kubenet
                                nonMasqueradeCIDR: 100.64.0.0/10
                            +   podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
                            -   podInfraContainerImage: k8s.gcr.io/pause-amd64:3.0
                                podManifestPath: /etc/kubernetes/manifests
                            -   requireKubeconfig: true

                              __EOF_CLUSTER_SPEC
                            ...
                              nodeLabels:
                                kops.k8s.io/instancegroup: nodes
                            + suspendProcesses: null
                              taints: null

                            ...
                              cat > kube_env.yaml << '__EOF_KUBE_ENV'
                              Assets:
                            + - ef979a00ba2f7bf4ee5023e82f94ced2d94c1726@https://storage.googleapis.com/kubernetes-release/release/v1.9.3/bin/linux/amd64/kubelet
                            - - 0f3a59e4c0aae8c2b2a0924d8ace010ebf39f48e@https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubelet
                            + - a27d808eb011dbeea876fe5326349ed167a7ed28@https://storage.googleapis.com/kubernetes-release/release/v1.9.3/bin/linux/amd64/kubectl
                            - - 36340bb4bb158357fe36ffd545d8295774f55ed9@https://storage.googleapis.com/kubernetes-release/release/v1.8.7/bin/linux/amd64/kubectl
                            - - 1d9788b0f5420e1a219aad2cb8681823fc515e7c@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
                            + - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
                            + - c6f310214f687b6c2f32e81c2a49235182950be3@https://kubeupv2.s3.amazonaws.com/kops/1.9.0/linux/amd64/utils.tar.gz
                            - - 42b15a0a0a56531750bde3c7b08d0cf27c170c48@https://kubeupv2.s3.amazonaws.com/kops/1.8.1/linux/amd64/utils.tar.gz
                              ClusterName: hello.k8s.local
                              ConfigBase: s3://state.hello.k8s.local/hello.k8s.local
                            ...
                              - s3://state.hello.k8s.local/hello.k8s.local/addons/bootstrap-channel.yaml
                              protokubeImage:
                            +   hash: 4bbfcc6df1c1c0953bd0532113a74b7ae21e0ded
                            -   hash: 0b1f26208f8f6cc02468368706d0236670fec8a2
                            +   name: protokube:1.9.0
                            -   name: protokube:1.8.1
                            +   source: https://kubeupv2.s3.amazonaws.com/kops/1.9.0/images/protokube.tar.gz
                            -   source: https://kubeupv2.s3.amazonaws.com/kops/1.8.1/images/protokube.tar.gz

                              __EOF_KUBE_ENV
                            ...


  ManagedFile/hello.k8s.local-addons-bootstrap
    Contents
                            ...
                                  selector:
                                    k8s-addon: kube-dns.addons.k8s.io
                            +     version: 1.14.9
                            -     version: 1.14.8
                                - id: k8s-1.6
                                  kubernetesVersion: '>=1.6.0'
                            ...
                                  selector:
                                    k8s-addon: kube-dns.addons.k8s.io
                            +     version: 1.14.9
                            -     version: 1.14.8
                                - id: k8s-1.8
                                  kubernetesVersion: '>=1.8.0'
                            ...
                                  selector:
                                    k8s-addon: dns-controller.addons.k8s.io
                            +     version: 1.9.0
                            -     version: 1.8.0
                                - id: k8s-1.6
                                  kubernetesVersion: '>=1.6.0'
                            ...
                                  selector:
                                    k8s-addon: dns-controller.addons.k8s.io
                            +     version: 1.9.0
                            -     version: 1.8.0
                                - id: v1.7.0
                                  kubernetesVersion: '>=1.7.0'
                            ...


  ManagedFile/hello.k8s.local-addons-dns-controller.addons.k8s.io-k8s-1.6
    Contents
                            ...
                                  k8s-addon: dns-controller.addons.k8s.io
                                  k8s-app: dns-controller
                            +     version: v1.9.0
                            -     version: v1.8.0
                                name: dns-controller
                                namespace: kube-system
                            ...
                                      k8s-addon: dns-controller.addons.k8s.io
                                      k8s-app: dns-controller
                            +         version: v1.9.0
                            -         version: v1.8.0
                                  spec:
                                    containers:
                            ...
                                      - --zone=*/*
                                      - -v=2
                            +         image: kope/dns-controller:1.9.0
                            -         image: kope/dns-controller:1.8.0
                                      name: dns-controller
                                      resources:
                            ...


  ManagedFile/hello.k8s.local-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
    Contents
                            ...
                                  k8s-addon: dns-controller.addons.k8s.io
                                  k8s-app: dns-controller
                            +     version: v1.9.0
                            -     version: v1.8.0
                                name: dns-controller
                                namespace: kube-system
                            ...
                                      k8s-addon: dns-controller.addons.k8s.io
                                      k8s-app: dns-controller
                            +         version: v1.9.0
                            -         version: v1.8.0
                                  spec:
                                    containers:
                            ...
                                      - --zone=*/*
                                      - -v=2
                            +         image: kope/dns-controller:1.9.0
                            -         image: kope/dns-controller:1.8.0
                                      name: dns-controller
                                      resources:
                            ...


  ManagedFile/hello.k8s.local-addons-kube-dns.addons.k8s.io-k8s-1.6
    Contents
                            ...
                                      - --logtostderr=true
                                      - --v=2
                            +         image: gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.2-r2
                            -         image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.1.2-r2
                                      name: autoscaler
                                      resources:
                            ...
                                      - name: PROMETHEUS_PORT
                                        value: "10055"
                            +         image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.9
                            -         image: k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
                                      livenessProbe:
                                        failureThreshold: 5
                            ...
                                      - -k
                                      - --cache-size=1000
                            +         - --dns-forward-max=150
                                      - --no-negcache
                                      - --log-facility=-
                            ...
                                      - --server=/in-addr.arpa/127.0.0.1#10053
                                      - --server=/in6.arpa/127.0.0.1#10053
                            +         image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.9
                            -         image: k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
                                      livenessProbe:
                                        failureThreshold: 5
                            ...
                                      - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
                                      - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
                            +         image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.9
                            -         image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
                                      livenessProbe:
                                        failureThreshold: 5
                            ...


  ManagedFile/hello.k8s.local-addons-kube-dns.addons.k8s.io-pre-k8s-1.6
    Contents
                            ...
                                      - --logtostderr=true
                                      - --v=2
                            +         image: gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.0.0
                            -         image: k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.0.0
                                      name: autoscaler
                                      resources:
                            ...
                                      - name: PROMETHEUS_PORT
                                        value: "10055"
                            +         image: gcr.io/google_containers/kubedns-amd64:1.9
                            -         image: k8s.gcr.io/kubedns-amd64:1.9
                                      livenessProbe:
                                        failureThreshold: 5
                            ...
                                    - args:
                                      - --cache-size=1000
                            +         - --dns-forward-max=150
                                      - --no-resolv
                                      - --server=127.0.0.1#10053
                                      - --log-facility=-
                            +         image: gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.14.9
                            -         image: k8s.gcr.io/k8s-dns-dnsmasq-amd64:1.14.8
                                      livenessProbe:
                                        failureThreshold: 5
                            ...
                                      - --v=2
                                      - --logtostderr
                            +         image: gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
                            -         image: k8s.gcr.io/dnsmasq-metrics-amd64:1.0
                                      livenessProbe:
                                        failureThreshold: 5
                            ...
                                      - --port=8080
                                      - --quiet
                            +         image: gcr.io/google_containers/exechealthz-amd64:1.2
                            -         image: k8s.gcr.io/exechealthz-amd64:1.2
                                      name: healthz
                                      ports:
                            ...


  RouteTable/hello.k8s.local
    Tags                     {KubernetesCluster: hello.k8s.local, Name: hello.k8s.local} -> {Name: hello.k8s.local, KubernetesCluster: hello.k8s.local, kubernetes.io/cluster/hello.k8s.local: owned, kubernetes.io/kops/role: public}

  SecurityGroup/masters.hello.k8s.local
    Tags                     {Name: masters.hello.k8s.local, KubernetesCluster: hello.k8s.local} -> {Name: masters.hello.k8s.local, KubernetesCluster: hello.k8s.local, kubernetes.io/cluster/hello.k8s.local: owned}

  SecurityGroup/nodes.hello.k8s.local
    Tags                     {KubernetesCluster: hello.k8s.local, Name: nodes.hello.k8s.local} -> {Name: nodes.hello.k8s.local, KubernetesCluster: hello.k8s.local, kubernetes.io/cluster/hello.k8s.local: owned}

Must specify --yes to apply changes

% kops update cluster --yes
Using cluster from kubectl context: hello.k8s.local

I0420 20:31:55.615730   47173 executor.go:91] Tasks: 0 done / 77 total; 31 can run
I0420 20:31:55.696294   47173 logging_retryer.go:60] Retryable error (RequestError: send request failed
caused by: Post https://ec2.us-west-2.amazonaws.com/: EOF) from ec2/DescribeKeyPairs - will retry after delay of 35ms
I0420 20:31:59.537746   47173 executor.go:91] Tasks: 31 done / 77 total; 26 can run
I0420 20:32:04.352538   47173 executor.go:91] Tasks: 57 done / 77 total; 18 can run
I0420 20:32:11.539050   47173 executor.go:91] Tasks: 75 done / 77 total; 2 can run
I0420 20:32:12.183580   47173 executor.go:91] Tasks: 77 done / 77 total; 0 can run
I0420 20:32:12.184122   47173 dns.go:153] Pre-creating DNS records
I0420 20:32:15.840376   47173 update_cluster.go:291] Exporting kubecfg for cluster
kops has set your kubectl context to hello.k8s.local

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster

kops rolling-update cluster コマンドでEC2インスタンスを再作成します。

% kops rolling-update cluster
Using cluster from kubectl context: hello.k8s.local

NAME            STATUS      NEEDUPDATE  READY   MIN MAX NODES
master-us-west-2a   NeedsUpdate 1       0   1   1   1
nodes           NeedsUpdate 3       0   3   3   3

Must specify --yes to rolling-update.

% kops rolling-update cluster --yes
Using cluster from kubectl context: hello.k8s.local

NAME            STATUS      NEEDUPDATE  READY   MIN MAX NODES
master-us-west-2a   NeedsUpdate 1       0   1   1   1
nodes           NeedsUpdate 3       0   3   3   3
I0420 20:38:41.164848   47267 instancegroups.go:157] Draining the node: "ip-172-20-39-230.us-west-2.compute.internal".
node "ip-172-20-39-230.us-west-2.compute.internal" cordoned
node "ip-172-20-39-230.us-west-2.compute.internal" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: etcd-server-events-ip-172-20-39-230.us-west-2.compute.internal, etcd-server-ip-172-20-39-230.us-west-2.compute.internal, kube-apiserver-ip-172-20-39-230.us-west-2.compute.internal, kube-controller-manager-ip-172-20-39-230.us-west-2.compute.internal, kube-proxy-ip-172-20-39-230.us-west-2.compute.internal, kube-scheduler-ip-172-20-39-230.us-west-2.compute.internal
pod "dns-controller-dcb5b7668-kjb5m" evicted
node "ip-172-20-39-230.us-west-2.compute.internal" drained
I0420 20:40:21.517156   47267 instancegroups.go:273] Stopping instance "i-09db1c4916f902ccc", node "ip-172-20-39-230.us-west-2.compute.internal", in group "master-us-west-2a.masters.hello.k8s.local".
I0420 20:45:23.226395   47267 instancegroups.go:188] Validating the cluster.
I0420 20:45:32.898059   47267 instancegroups.go:249] Cluster validated.
I0420 20:45:35.639303   47267 instancegroups.go:157] Draining the node: "ip-172-20-42-214.us-west-2.compute.internal".
node "ip-172-20-42-214.us-west-2.compute.internal" cordoned
node "ip-172-20-42-214.us-west-2.compute.internal" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-ip-172-20-42-214.us-west-2.compute.internal; Ignoring DaemonSet-managed pods: prometheus-node-exporter-hp78d
pod "kube-dns-6c4cb66dfb-zpbg8" evicted
pod "heapster-heapster-697757c69d-7q72n" evicted
...
node "ip-172-20-42-214.us-west-2.compute.internal" drained
I0420 20:47:14.245433   47267 instancegroups.go:273] Stopping instance "i-08698814f70aa43bb", node "ip-172-20-42-214.us-west-2.compute.internal", in group "nodes.hello.k8s.local".
I0420 20:51:16.646042   47267 instancegroups.go:188] Validating the cluster.
I0420 20:51:30.401590   47267 instancegroups.go:249] Cluster validated.
I0420 20:51:30.401634   47267 instancegroups.go:157] Draining the node: "ip-172-20-40-170.us-west-2.compute.internal".
node "ip-172-20-40-170.us-west-2.compute.internal" cordoned
node "ip-172-20-40-170.us-west-2.compute.internal" cordoned
WARNING: Ignoring DaemonSet-managed pods: prometheus-node-exporter-mzcb9; Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-ip-172-20-40-170.us-west-2.compute.internal; Deleting pods with local storage: kubernetes-dashboard-778c8bcdb6-56mgh
pod "kube-dns-6c4cb66dfb-tbnpw" evicted
pod "kube-dns-autoscaler-f4c47db64-znjhn" evicted
...
node "ip-172-20-40-170.us-west-2.compute.internal" drained
I0420 20:53:27.672263   47267 instancegroups.go:273] Stopping instance "i-0bf2956226b308e8e", node "ip-172-20-40-170.us-west-2.compute.internal", in group "nodes.hello.k8s.local".
I0420 20:57:30.211080   47267 instancegroups.go:188] Validating the cluster.
I0420 20:57:35.694808   47267 instancegroups.go:249] Cluster validated.
I0420 20:57:35.694846   47267 instancegroups.go:157] Draining the node: "ip-172-20-33-12.us-west-2.compute.internal".
node "ip-172-20-33-12.us-west-2.compute.internal" cordoned
node "ip-172-20-33-12.us-west-2.compute.internal" cordoned
WARNING: Deleting pods with local storage: grafana-5d74df4b45-b9srz; Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-ip-172-20-33-12.us-west-2.compute.internal; Ignoring DaemonSet-managed pods: prometheus-node-exporter-95wj6
pod "heapster-heapster-697757c69d-mzxxh" evicted
pod "kube-dns-6c4cb66dfb-mbd59" evicted
...
node "ip-172-20-33-12.us-west-2.compute.internal" drained
I0420 20:59:46.310060   47267 instancegroups.go:273] Stopping instance "i-0e3622a73e8046b63", node "ip-172-20-33-12.us-west-2.compute.internal", in group "nodes.hello.k8s.local".
I0420 21:03:47.697124   47267 instancegroups.go:188] Validating the cluster.
I0420 21:03:53.581898   47267 instancegroups.go:249] Cluster validated.
I0420 21:03:53.581958   47267 rollingupdate.go:193] Rolling update completed for cluster "hello.k8s.local"!