* Managed and maintained a fleet of approximately 30,000 Linux servers across 200+ data centers worldwide, all dual-stacked with both IPv4 and IPv6
* Oversaw management and maintenance of 30,000 Linux servers across 200+ data centers worldwide
* Over my 12 years of employment there, my team and I upgraded all servers from Ubuntu 10.04 to 22.04
* Over my 12 years of employment there, my team and I upgraded all servers from Ubuntu 10.04 to 22.04
* Went through several generations of Dell servers as we upgraded and installed new POPs, from R510, R710, R720 R730's, and R610, R620, R630 and R640's. All servers were managed and provisioned remotely through iDRAC access
* Went through several generations of Dell servers as we upgraded and installed new POPs, from R510, R710, R720 R730's, and R610, R620, R630 and R640's. All servers were managed and provisioned remotely through iDRAC access
* Deployed approximately 100 POPs with an average of 300 servers per POP in data centers across the world
* Deployed approximately 100 POPs with an average of 300 servers per POP
* Maintained the lifecycle management of each server by making sure all firmware (iDRAC, BIOS, Lifecycle Controller, NIC's, SSD's etc.) was updated every time we provisioned or reprovisioned a server
* Maintained the lifecycle management of each server by making sure all firmware (iDRAC, BIOS, Lifecycle Controller, NIC's, SSD's etc.) was updated every time we provisioned or reprovisioned a server
* Used fully automatic installation (fai-project.org) to build all of our servers via PXE booting. Once the OS was laid down, we'd use a configuration management tool to finalize the server configuration
* Used fully automatic installation (fai-project.org) to build all of our servers via PXE booting. Once the OS was laid down, we'd use a configuration management tool to finalize the server configuration
* SaltStack was introduced as a configuration management tool to replace dozens of shell scripts that configure our servers. Committed many changes to salt states and pillars as needed to make appropriate changes to the fleet
* SaltStack was introduced as a configuration management tool to replace dozens of shell scripts that configure our servers. Committed many changes to salt states and pillars as needed to make appropriate changes to the fleet
...
@@ -56,9 +56,9 @@ Resume
...
@@ -56,9 +56,9 @@ Resume
* Managed and maintained appdoximately 700 VMHosts and approcimately 2000 VM's
* Managed and maintained appdoximately 700 VMHosts and approcimately 2000 VM's
* Used Sphinx-Docs and GitLab CI/CD to build our documentation. This allowed us to migrate off of Confluence and inroduced the ability to use docs-as-code where we kept a git repo with all our documentation. While anyone was allowed to fork, clone, make a feature branch and send a merge request to the main documentation project, we enforced peer review requirements by requiring one approval from our US team and one from our London team. When the merge was approved the CI/CD process would build and publish the documentation to GitLab Pages
* Used Sphinx-Docs and GitLab CI/CD to build our documentation. This allowed us to migrate off of Confluence and inroduced the ability to use docs-as-code where we kept a git repo with all our documentation. While anyone was allowed to fork, clone, make a feature branch and send a merge request to the main documentation project, we enforced peer review requirements by requiring one approval from our US team and one from our London team. When the merge was approved the CI/CD process would build and publish the documentation to GitLab Pages
* Used a validation system (smoke checks) that would continuously run and check the health of the servers. Smoke checks were simple bash or python scripts that ran every minute and sent alerts to the application owners of the servers. For example we'd check disk space, link speed, CPU utilization and application health and report issues through Slack or other means
* Used a validation system (smoke checks) that would continuously run and check the health of the servers. Smoke checks were simple bash or python scripts that ran every minute and sent alerts to the application owners of the servers. For example we'd check disk space, link speed, CPU utilization and application health and report issues through Slack or other means
* Worked with the network team on a 80/20 basis for a couple months where I would setup peering arrangements with other ISPs also located at the same data center. At first we would peer via the data centers internet exchange (IX) and if the traffic grew to 1gbps or greater we would request to peer with them via Private Network Interconnect (PNI) where we would plug our router directly into theirs with a 10gb link
* Worked with the network team on a 80/20 basis for a couple months where I learned to establish peering arrangements with other ISPs also located at the same data center. At first we would peer via the data centers internet exchange (IX) and if the traffic grew to 1gbps or greater we would request to peer with them via Private Network Interconnect (PNI) where we would connect our router directly into theirs with a 10gb link
* Prepared a course on how to use Kubernetes for my team, introducing them to the core components such as creating images, pods, deployments, services, configmaps and adjusting replica counts
* Prepared a course on how to use Kubernetes for my team, introducing them to the core components such as creating images, pods, deployments, services, configmaps, container registries and adjusting replica counts
* As a team we took great pride in our work in helping others. We had a self-imposed 1-2 minute SLA for responding to people in Slack
* As a team we took great pride in our work in helping others. We had a self-imposed 5 minute SLA for responding to people in Slack and were well known as the best place to go to for support
* Maintained the Linux mail relays running Postfix
* Maintained the Linux mail relays running Postfix
* Worked with auditors on yearly PCI compliance projects
* Worked with auditors on yearly PCI compliance projects
