Comprehensive Study Notes on Terraform, Automation, and Onboarding

Terraform and Automation

• Terraform Overview

  • Controls infrastructure changes and management.
  • Not, in itself, an automation tool.
  • Requires an automation pipeline to gather details and deploy resources.

• Deployment and Automation Pipeline

  • Current pipelines manage base Google Cloud Platform (GCP) projects, but more advanced automation is required for additional functionalities.
  • Management of add-ons in Terraform necessitates manual intervention and is termed a 'semi-manual process'.
  • Key clarification:
  • "Manual creation of resources via Terraform" means using Terraform scripts while still necessitating manual input from users rather than using a GUI.

• Vertex AI Deployment Issue

  • The Vertex AI deployment has been executed twice without templatization or standardized deployment definitions.
  • Standardized deployment templates are needed to streamline the process.
  • Once a template is defined, automation can simplify changes to deployment by merely toggling options via checkboxes, ensuring modular deployment.

• Identification of Automated and Non-automated Pieces

  • Some components, such as buckets, are fully automated.
  • Non-automated components require ticket generation for manual processes.
  • Example scenario: Kimberly must open a ticket linking to a Jenkins job to execute specific actions.
  • Non-automated actions necessitate manual intervention despite being created via Terraform.

• Terminology Consistency

  • It is crucial to maintain clear terminology to differentiate between actions done "by hand" and those executed via Terraform.

Data Collection and Onboarding Process

• Initial Steps to Streamline Onboarding

  • Identifying necessary data upfront will improve the efficiency of onboarding requests.
  • Eliminate unnecessary questionnaires and forms to simplify the process.
  • Recommended inclusion of the following key data in onboarding forms:
  • Solution type (e.g., DLF, Analytics, Snowflake)
  • Cloud platform preference (AWS or GCP)
  • Client name, cost center, solution owner, and technical lead names.

• Role-Based Access Control (RBAC) Groups

  • Initial members for RBAC groups should be established during the onboarding call.
  • Explore with ServiceNow admins the feasibility of increasing sync frequency of RBAC catalogs for timely updates.

• Ticket System Updates

  • Ticket submissions must clearly define what resources are needed and any automation required in the onboarding process.
  • Collect information that will be essential to run the automation pipeline effectively, including:
  • Solution types and required resource configurations.

Pixie and CMDB Revisions

• Pixie Page Updates

  • Update the CMDB to reflect changes in definitions and keep both the Pixie and CMDB up-to-date.
  • Ensure that security approvers are included in this process.

• Templates and Documentation

  • Templates for GCP operations must be updated to include links to working Jenkins pipelines, which should shift to the new Jenkins system (Buildasaurus).
  • Establish a checklist for migrating jobs, ensuring proper tracking of updated links and resources.

Databricks Integration Consideration

• Integration Challenges

  • There needs to be clarity on what specific templates should be used for onboarding resource requests involving Databricks.
  • Clarification needed on whether Databricks solutions fall under a particular analytics category or requires a standalone template.

• Action Items for Databricks

  • Gather more information about which templates under Snowflake Analytics or others category Databricks should be associated with.
  • Confirm approval processes specific to Databricks deployments before provisioning.

Jenkins and Automation Efforts

• Monitoring Jenkins Pipeline Migrations

  • Regularly check that Jenkins pipelines transition properly to the new system.
  • Maintain rigorous monitoring to ensure that broken links and old job definitions are purged or updated accordingly.

• Collaboration and Communication Efficiency

  • Maintain clear communication channels among team members regarding operational status and open issues.
  • Ensure consistent updates and ownership of documentation needed to facilitate onboarding and resource management.