2.3-Resource Management in Google Cloud

1-Resource Management

Overview

• Resources in Google Cloud are billable, so managing them is important for controlling costs. Effective resource management ensures you're only paying for what you need and prevents unnecessary expenditures.

• There are methods for controlling access to resources, and quotas limit consumption. Access control ensures that only authorized personnel can modify or use resources, while quotas prevent over-provisioning and potential waste.

• Default quotas can be raised upon request, but they act as a checkpoint to verify resource consumption intentions. This verification step helps ensure that any increase in resource usage aligns with your planned activities and business needs.

Agenda

• Resource Manager: Overview of how to manage resources hierarchically. Learn how to organize your resources into a structured hierarchy using projects, folders, and organizations.

• Quotas: Limits on resource consumption. Understand the different types of quotas and how they impact your ability to create and use resources.

• Labels: Organizing resources with key-value pairs to filter and analyze them. Discover how labels can help you categorize and track your resources for better visibility and management.

• Billing: Configuring budgets and alerts to track and manage spending. Set up budgets and alerts to proactively monitor your Google Cloud spending and prevent unexpected charges.

• Lab: Examining billing data with BigQuery to analyze resource consumption. Gain hands-on experience with BigQuery to analyze your billing data and identify opportunities for cost optimization.

Resource Manager

• The Resource Manager lets you hierarchically manage resources by project, folder, and organization. This hierarchy allows you to apply policies and controls consistently across your Google Cloud environment.

• IAM (Identity and Access Management) policies contain roles and members and are set on resources. IAM policies define who has access to what resources and what actions they can perform.

• Resources inherit policies from their parent. This inheritance simplifies policy management and ensures consistent access controls.

• Resource policies are a union of parent and resource if an IAM allow policy is associated. This means that the effective policy for a resource is the combination of the policies applied to the resource itself and its parent resources.

• If an IAM deny policy is associated with the resource, then the policy can prevent certain principals from using certain permissions, regardless of the roles they’re granted. Deny policies provide a way to explicitly restrict access to resources, even if a user has been granted those permissions through other means.

• IAM policies are inherited top-to-bottom, billing is accumulated from the bottom up. Policies flow down the resource hierarchy, while costs roll up to the billing account.

• Resource consumption is measured in quantities like rate of use or time, number of items, or feature use. Understanding how resource consumption is measured is essential for accurate cost tracking and optimization.

• A resource belongs to only one project, and a project accumulates the consumption of all its resources. Projects serve as the top-level container for your resources and are associated with a billing account.

• Each project is associated with one billing account, and an organization contains all billing accounts. The organization provides a central point for managing billing and access control across your Google Cloud environment.

• The organization node is the root node for all Google Cloud resources. It represents your company or organization and provides a way to manage resources and policies at a high level.

• Projects accumulate the consumption of all their resources.

• Track resource and quota usage.

• Enable billing.

• Manage permissions and credentials.

• Enable services and APIs.

• Projects use three identifying attributes:

  • Project Name

  • Project Number

  • Project ID, also known as Application ID

Identifying Attributes of a Project

• Project Name: A human-readable way to identify projects but not used by Google APIs. Choose descriptive names that make it easy to identify your projects.

• Project Number: Automatically generated by the server and assigned to a project. This number is a unique identifier for your project.

• Project ID: A unique ID generated from the project name. The Project ID is used to identify your project in API calls and other interactions with Google Cloud services.

Resource Hierarchy

• Resources are categorized as global, regional, or zonal. Understanding the scope of your resources is important for planning your deployments and managing costs.

• Global:

  • Images

  • Snapshots

  • Networks

• Regional:

  • External IP addresses

• Zonal:

  • Instances

  • Disks

• Regardless of the type, each resource is organized into a project. This organization ensures that all resources are associated with a billing account and can be managed consistently.

• This enables each project to have its own billing and reporting.

2-Quotas

• All resources in Google Cloud are subject to project quotas or limits. These quotas are in place to prevent misuse and ensure fair access to resources for all users.

• Types of quotas:

  • How many resources you can create per project e.g., 15 VPC networks/project.

  • How quickly you can make API requests in a project (rate limits) e.g., 5 admin actions/second (Spanner).

  • How many resources you can create per region e.g., 24 CPUs region/project.

• Quotas can be increased via the Quotas page in the Google Cloud console or by submitting a support ticket. If you need more resources than the default quotas allow, you can request an increase.

• Project quotas:

  • Prevent runaway consumption in case of an error or malicious attack.

  • Prevent billing spikes or surprises.

  • Forces sizing consideration and periodic review.

• Quotas are the maximum amount of resources you can create for that resource type as long as those resources are available. Quotas do not guarantee that resources will be available at all times. Resource availability can vary depending on demand and other factors.

3-Labels

• Labels are a utility for organizing Google Cloud resources. By applying labels to your resources, you can easily group and filter them for management and analysis.

• Labels are key-value pairs that you can attach to resources like VMs, disks, snapshots, and images. Each label consists of a key and a value, both of which are user-defined strings.

• Labels can be managed using the Google Cloud console, gcloud, or the Resource Manager API. You can use any of these tools to create, update, and delete labels.

• Each resource can have up to 64 labels.

• Example uses of labels:

  • Inventory

  • Filter resources

  • In scripts

    • Help analyze costs

    • Run bulk operations

• Use labels for:

  • Team or Cost Center: team:marketing, team:research

  • Components: component:redis, component:frontend

  • Environment or stage: environment:prod, environment:test

  • Owner or contact: owner:gaurav, contact:opm

  • State: state:inuse, state:readyfordeletion

Labels vs. Network Tags

• Labels:

  • A way to organize resources across Google Cloud.

  • Can be applied to disks, images, snapshots, etc.

  • User-defined strings in key-value format.

  • Propagated through billing.

• Network Tags:

  • Applied to instances only.

  • User-defined strings.

  • Primarily used for networking (applying firewall rules).

4-Billing

• The consumption of all resources under a project accumulates into one billing account. This aggregation simplifies billing and makes it easier to track costs.

• Budgets and email alerts:

  • To help with project planning and controlling costs, you can set a budget. Setting a budget helps you stay within your spending limits and avoid unexpected charges.

  • Setting a budget lets you track how your spend is growing toward that amount. You can monitor your spending against your budget in the Google Cloud console.

  • You can set the budget at a specific amount or match it to the previous month's spend. Setting a budget based on previous spending can help you maintain consistent cost control.

  • After you determine your budget amount, you can set the budget alerts. Budget alerts notify you when your spending exceeds a certain percentage of your budget.

  • These alerts send emails to Billing Admins after spend exceeds a percentage of the budget or a specified amount. You can customize the alert thresholds to suit your needs.

  • You can even choose to send an alert when the spend is forecasted to exceed the percentage of the budget amount by the end of the budget period. This forecasting can help you take proactive steps to reduce spending before it exceeds your budget.

  • In addition to receiving an email, you can use Pub/Sub notifications to programmatically receive spend updates about this budget. Pub/Sub notifications allow you to integrate your budget alerts with other systems and automate cost management tasks.

  • You could even create a Cloud Run function that listens to the Pub/Sub topic to automate cost management. This automation can help you respond quickly to unexpected spending patterns.

• Labels can help you optimize Google Cloud spend. By applying labels to your resources, you can break down your spending by team, project, or environment.

• For example, you could label VM instances that are spread across different regions. This labeling allows you to see how much you're spending on VMs in each region.

• I recommend labeling all your resources and exporting your billing data to BigQuery to analyze your spend. Exporting your billing data to BigQuery allows you to perform complex queries and gain deeper insights into your spending patterns.

• BigQuery is Google’s scalable, fully managed enterprise data warehouse with SQL and fast response times. BigQuery can handle large volumes of data and provide query results quickly.

Visualizing Spend with Looker Studio (p20 for charts)

• You can even visualize spend over time with Looker Studio. Visualizing your spending data can help you identify trends and patterns that might not be apparent in raw data.

• Looker Studio turns your data into informative dashboards and reports that are easy to read, easy to share, and fully customizable. You can use Looker Studio to create a variety of reports and dashboards to track your Google Cloud spending.

• For example, you can slice and dice your billing reports using your labels. This slicing and dicing allows you to see how much you're spending on different teams, projects, or environments.

Lab: Examining Billing Data with BigQuery

• In this lab, you will sign in to BigQuery and create a dataset. Creating a dataset in BigQuery is the first step in importing and analyzing your billing data.

• In this dataset, you will create a table by importing billing data that is stored in a Cloud Storage bucket. Importing your billing data into BigQuery allows you to query and analyze it using SQL.

• Next, you will run simple queries on the imported data, and then you will run more complex queries on a larger dataset. Experimenting with different queries can help you gain a better understanding of your spending patterns.

• In this lab, you imported billing data into BigQuery that had been exported as a CSV file.

• You first ran a simple query on that data.

• Next, you accessed a shared dataset containing more than 22,000 records of billing information.

• You then ran a variety of queries on that data to explore how you can use BigQuery to gain insight into your resources’ billing consumption.

• If you use BigQuery on a regular basis, you'll start to develop your own queries for searching out where resources are being consumed in your application. Over time, you'll become more proficient at using BigQuery to analyze your Google Cloud spending.

• You can also monitor changes in resource consumption over time. Monitoring changes in resource consumption can help you identify trends and potential cost savings opportunities.

• This kind of analysis is an input to capacity planning and can help you determine how to scale up your application to meet growth or scale down your application for efficiency.

Review

• In this module, we covered the Cloud Resource Manager and went into quotas, labels, and billing.

• Then we examined billing data with BigQuery in a lab.

• Reporting is an important part of resource management. Regularly reviewing reports on resource consumption and billing data will empower stakeholders to make informed decisions and optimize cloud usage.

• You can generate reports to track consumption and to establish accountability. These reports can be customized to show specific metrics or time periods, providing insights tailored to the needs of different teams or departments.

• A key principle in Google Cloud is transparency, and that means it's straightforward to access and process consumption data, as you observed in this module. Google Cloud provides tools