Module 4: Google Cloud Core Services
Identity and Access Management (IAM)
Identity and Access Management
Identity and Access Management (IAM) in Google Cloud is a system that defines who can perform what actions on which resources. It helps control permissions for users, groups, or applications.
Who: Refers to the identity, such as a person, group, or application.
What: Refers to the specific actions or privileges that can be performed (e.g., view, edit, delete).
Resource: This is the Google Cloud service or asset (e.g., Compute Engine, Cloud Storage).
For instance, if you’re given the "Compute Viewer" role, you can only view Compute Engine resources without accessing the data stored on them.
Cloud IAM Components:
IAM Policies: Define who has access to what, based on roles.
Resource Hierarchy: Google Cloud organizes resources hierarchically:
Organization: Represents your company and is the root of the hierarchy.
Folders: Child resources under the organization, could represent departments.
Projects: Represent trust boundaries within your organization, where services have the same trust level.
Resources: These are individual services or assets under a project.
Roles granted at higher levels (like organization or folder) are inherited by the resources beneath them.
Organization
The organization node in Google Cloud is the root of the GCP resource hierarchy and has several important roles:
Organization Admin: This role gives full administrative control over all resources in the organization, useful for auditing and managing all aspects of the organization.
Project Creator: This role allows a user to create projects within the organization. If granted at the organization level, it applies to all projects within the organization.
The organization resource is linked to a G Suite or Cloud Identity Account.
When a user with one of these accounts creates a GCP project, an organization resource is automatically created for them.
The G Suite or Cloud Identity super admins are critical, as they have extensive control over the organization and resources, including assigning the organization admin role.
Key Roles:
G Suite/Cloud Identity Super Admin: Controls the lifecycle of the G Suite or Cloud Identity account and the organization resource.
Organization Admin: Manages IAM policies, defines the resource hierarchy, and delegates roles over critical components like networking and billing, adhering to the principle of least privilege.
Folders in GCP:
Folders act as sub-organizations, providing isolation boundaries between projects.
They can represent different entities or departments within an organization (e.g., department X, department Y) and allow delegation of administration rights, like assigning department heads full ownership of resources within their department.
Folders allow for more granular control over resource access.
Other roles:
Organization Node Viewer: Provides read-only access to all resources within the organization.
Folder Roles:
Admin: Full control over folders.
Creator: Allows browsing the hierarchy and creating folders.
Viewer: Provides view access to folders and projects under a resource.
Project Roles:
Creator: Allows the user to create new projects and automatically make them the owner.
Deleter: Grants deletion privileges for projects.
These roles and the inheritance system allow for organized, flexible, and secure management of resources within an organization.
Roles
Basic Roles:
Owner: Full administrative access, including the ability to add/remove members and delete projects.
Editor: Modify and delete resources, allowing actions like deploying applications and configuring resources.
Viewer: Read-only access to resources.
These roles are concentric, meaning:
Owner includes Editor and Viewer permissions.
Editor includes Viewer permissions.
Additional roles like the Billing Administrator allow for billing management without access to modify project resources.
Predefined Roles:
These roles are more specific to Google Cloud services and grant granular access to particular resources, helping prevent unnecessary access to unrelated resources.
Compute Engine roles:
Compute Admin: Full control of all Compute Engine resources, allowing every action for any Compute Engine resource.
Network Admin: Permissions to manage network resources (but not firewall rules or SSL certificates) and view ephemeral IP addresses of instances.
Storage Admin: Permissions to manage disks, images, and snapshots, without full project editing rights.
Predefined roles are collections of permissions that simplify management.
These permissions correspond to API classes and methods (e.g., starting a Compute Engine instance involves permissions like
compute.instance.start).
Custom Roles:
If predefined roles are too broad or don’t meet specific needs, custom roles can be created.
These allow you to assign precise permissions, following the least privilege model.
For example, a custom role could allow users to start and stop Compute Engine virtual machines but prevent them from reconfiguring them, providing a more fine-grained access control.
Custom roles are ideal for aligning permissions with specific job functions, ensuring users only have the minimal access required to perform their duties.
Members
Types of Members:
Google Accounts: Represent developers, administrators, or any user interacting with Google Cloud. Can include gmail.com or other email domains.
Service Accounts: Belong to applications, not individuals. Used when running code on Google Cloud to specify the account under which the code runs.
Google Groups: Collections of Google accounts and service accounts. Used for applying access policies to groups, simplifying access management.
Google Workspace Domains: Represent a virtual group of accounts within an organization's Workspace, such as username@example.com.
Cloud Identity Domains: Similar to Workspace but without Workspace's collaboration products. Used by Google Cloud customers for user management.
IAM (Identity and Access Management):
IAM defines who can do what on which resource through policies made up of bindings, roles, and members.
Roles: Named lists of permissions, defining access to resources.
Policy Hierarchy: IAM policies inherit from parent resources. Parent policies can override more restrictive child policies.
Principle of Least Privilege: Always select the minimum permissions required for tasks to reduce risk.
Recommender: Suggests role changes to remove excess permissions and improve security.
Access Control:
Allow Policies: Bind members (users or service accounts) to roles, granting access to Google Cloud resources.
Deny Policies: Prevent certain principals from using specific permissions, checked before allow policies.
IAM Conditions: Conditional, attribute-based access control. Grants access only if specific conditions are met (e.g., temporary access, location-based).
Organization Policies:
Define restrictions at the organization node and are inherited by all descendants. Only users with the organization policy admin role can create exceptions.
Synchronizing External Directories:
Google Cloud Directory Sync: Syncs users and groups from Active Directory or LDAP with Cloud Identity, without modifying existing directories.
Single Sign-On (SSO): Allows users to authenticate with their own system. Google Cloud supports SSO with SAML2, or through third-party systems like ADFS, Ping, or Okta.
Creating Google Accounts Without Gmail:
Users can create Google accounts without using Gmail, just for accessing Google Cloud services.
Service accounts
A service account is an account for applications, not individuals, enabling service-to-service interactions without user credentials.
Used for seamless API authentication without embedding secret keys in code or instances.
Types:
Built-in:
Compute Engine Default Service Account: Auto-created with the Editor role (
project-number-compute@developer.gserviceaccount.com).Google Cloud APIs Service Account: Used for internal Google processes (
project-number@cloudservices.gserviceaccount.com).
Custom: User-created with flexible roles and access scopes but requiring more management.
Authentication Process:
Applications authenticate via Google Authorization Server, receiving access tokens with specific scopes (e.g., read-only or read-write).
Scopes: Legacy method to specify permissions; now replaced by IAM roles.
IAM Roles for Service Accounts:
Roles define what permissions service accounts have (e.g., InstanceAdmin for managing VMs).
Service Account User role allows users/groups to act as the service account and access its resources.
Best Practices:
Assign specific service accounts to VMs for microservices.
Avoid over-granting permissions with the Service Account User role.
Service Account Keys:
Google-Managed Keys: Automatically managed, rotated regularly.
User-Managed Keys: Created manually for external use but require user responsibility for security, rotation, and recovery.
Up to 10 keys per account; use alternatives like short-lived credentials when possible.
Key Management:
Use IAM API, gcloud CLI, or Google Cloud Console.
Google does not store user-managed private keys; loss is irreversible.
Organization restriction
Purpose:
Prevent data exfiltration via phishing or insider attacks.
Restrict access to resources within authorized Google Cloud organizations only.
Setup:
Managed Devices: Governed by organizational policies, used by employees to access resources.
Egress Proxy Configuration:
Configured by the egress proxy administrator to add organization restrictions headers to all requests from managed devices.
Blocks access to unauthorized Google Cloud organizations.
Functionality:
Google Cloud inspects requests for the organization restrictions header and permits or denies access based on the organization being accessed.
Use Cases:
Restrict employees to access resources only within your Google Cloud organization.
Allow employees to access vendor resources in specific external organizations.
Enable employees to read from Cloud Storage resources within authorized organizations.
IAM best practices
Resource Hierarchy & Inheritance:
Leverage projects to group resources sharing the same trust boundary.
Check policies on resources and account for inheritance.
Apply the principle of least privilege when assigning roles.
Regularly audit policies via Cloud audit logs and memberships of groups in policies.
Granting Roles to Groups:
Assign roles to groups rather than individuals for easier management.
Update group membership instead of changing Cloud IAM policies.
Audit group membership and control ownership of groups used in Cloud IAM policies.
Use multiple groups for finer access control:
Example: Separate groups for network admins, read-write roles, and read-only roles for Cloud Storage buckets.
Groups can exist for job roles or role assignments.
Service Accounts:
Be cautious when granting the Service Account User role (it provides access to all resources the service account can access).
Use a clear and standardized naming convention for service account display names.
Implement key rotation policies, audit keys using the
serviceAccount.keys.listmethod, and monitor for potential misuse.
Cloud Identity-Aware Proxy (Cloud IAP):
Establish a central authorization layer for HTTPS-accessed applications.
Use application-level access controls instead of relying on network firewalls or VPNs.
Cloud IAP ensures:
Resources can only be accessed by users/groups with the appropriate Cloud IAM role.
Authentication and fine-grained authorization checks are performed at the application level.
Storage and database services
Cloud Storage: A scalable object storage service for various use cases like serving website content, backups, and distributing data.
Key Features:
Scalable to exabytes of data.
Low latency (milliseconds to first byte).
High availability.
Single API across storage classes.
Structure:
Data is stored in buckets (globally unique names, not nested).
Objects (e.g., files) are stored within buckets and inherit bucket storage classes unless specified otherwise.
Storage Classes:
Standard: High-cost, for frequently accessed ("hot") data. No retrieval cost or minimum storage duration.
Nearline: Low-cost for infrequent access, 30-day minimum storage duration, and retrieval costs.
Coldline: Very low cost, 90-day minimum duration, higher retrieval costs.
Archive: Lowest cost, 365-day minimum duration, high retrieval cost, but data available in milliseconds.
Location Types:
Multi-region: Multiple geographic places (geo-redundant).
Dual-region: Specific pairs of regions.
Region: Single geographic location.
Access Control:
IAM Roles: Manage access at project, bucket, and object levels.
ACLs (Access Control Lists): Finer control, max 100 entries, define scope (who) and permission (what).
Signed URLs: Grant temporary access without requiring a Google account.
Object Lifecycle Management: Automates transitioning objects between storage classes to optimize costs.
Cloud storage features
Key Features of Cloud Storage
Encryption Options:
Customer-Supplied Encryption Keys (CSEK): Lets you use your own encryption keys instead of Google-managed ones.
Data Management Features:
Object Lifecycle Management:
Automates actions like deleting or archiving objects based on custom rules.
Examples:
Downgrade objects older than 1 year to Coldline Storage.
Delete objects created before a specific date.
Keep only the 3 most recent versions of objects in a bucket.
Note: Changes in lifecycle rules take up to 24 hours to take effect.
Object Versioning:
Enables keeping multiple versions of objects.
Useful for restoring deleted or overwritten objects.
Turning it off stops accumulating new versions but retains existing ones.
Google recommends Soft Delete over Object Versioning for protecting against accidental deletion.
Soft Delete:
Retains deleted/overwritten objects for a default of 7 days (can be configured up to 90 days).
After the retention period, objects are permanently deleted.
Retention Policies:
Object Retention Lock:
Sets minimum retention durations to meet regulatory or compliance requirements.
Can permanently prevent reduction or removal of retention time.
Synchronization and Notifications:
Directory Synchronization: Syncs VM directories with buckets.
Object Change Notifications: Configured using Pub/Sub.
Data Transfer Services
Transfer Appliance:
Physical hardware for securely migrating large data volumes (up to 1PB).
Storage Transfer Service:
High-performance import from sources like Cloud Storage buckets, Amazon S3, or HTTP/HTTPS.
Offline Media Import:
Uploads data via third-party providers using physical media like hard drives or tapes.
Global Consistency
Strong Global Consistency:
Ensures reliable results for operations like creating, reading, updating, or deleting objects:
No 404 Not Found errors for successful uploads or metadata updates.
Immediate reflection of changes in bucket and object listings.
Choosing a storage class
Storage Class Decision Tree:
Archive: Use if data is read < once per year.
Coldline: Use if data is read < once per 90 days.
Nearline: Use if data is read < once per 30 days.
Standard: Use for frequent reads and writes.
Location Types:
Region: Optimizes latency and bandwidth for same-region data consumers (e.g., analytics pipelines).
Dual-region: Balances performance and geo-redundancy for higher availability.
Multi-region: Suitable for serving geographically distributed data consumers and ensures high availability.
Autoclass Feature:
Automatically transitions objects between storage classes based on access patterns.
Objects start in Standard storage regardless of specified storage class in the request.
Moves less-accessed data to colder storage and frequently accessed data to Standard storage.
Advantages: No early deletion, retrieval, or transition charges.
Filestore
Filestore Overview:
A managed file storage service providing a file system interface and a shared file system for data.
Compatible with NFSv3 clients and supports enterprise applications without needing special plug-ins.
Offers independent tuning of performance and capacity, ensuring fast and predictable performance.
Key Features:
Scale-out performance and hundreds of terabytes of capacity.
File locking for collaboration.
No client-side software installation required.
Use Cases:
Enterprise Application Migration: Supports on-premises applications requiring a file system interface as they migrate to the cloud.
Media Rendering:
Allows Compute Engine instances to share file systems for collaborative visual effects workflows.
Scales with rendering workloads across multiple machines.
Electronic Design Automation (EDA):
Handles batch workloads across thousands of cores with large memory needs.
Provides universal file accessibility for manufacturing customers.
Data Analytics:
Offers low latency for complex financial modeling or environmental data analysis.
Instances can grow/shrink with performance needs.
Genome Sequencing:
Supports massive data sets with speed, scalability, and security for scientific research.
Provides predictable pricing.
Web Content Hosting:
Used by web developers and hosting providers for WordPress hosting and serving web content.
Benefits:
Eliminates time spent loading/offloading data.
Provides persistent and shareable storage for high-performance workloads.
Cloud SQL
Cloud SQL Overview:
Fully managed SQL database service supporting MySQL, PostgreSQL, and SQL Server.
Automatically applies patches and updates but retains native user administration.
Compatible with tools like Cloud Shell, App Engine, Google Workspace scripts, and external apps (e.g., SQL Workbench, Toad).
Key Features:
High performance with up to 64 TB storage, 60,000 IOPS, and 624 GB RAM.
Scales to 96 processor cores and supports read replicas for scaling out.
Supports major SQL versions (MySQL 5.6–8.0, PostgreSQL 9.6–15, SQL Server 2017/2019 editions).
High Availability (HA):
Primary and standby instances use synchronous replication.
Automatic failover ensures service continuity during zone/instance failures.
Backups and Recovery:
Automated and on-demand backups with point-in-time recovery.
Import/export support for mysqldump and CSV files.
Scaling:
Vertical scaling (requires restart) and horizontal scaling with read replicas.
For extensive horizontal scalability, consider Cloud Spanner.
Connection Types:
Private IP: For high performance and security when connecting from the same project/region.
Cloud SQL Auth Proxy: Recommended for external connections; handles authentication, encryption, and key rotation.
SSL Connection: For manual control with certificate management.
Unencrypted Connection: Authorize specific IPs for external access (least secure).
Decision Tree for Relational Data Storage:
Memorystore: In-memory data storage for microsecond response times (e.g., gaming, real-time analytics).
BigQuery: For relational data used primarily in analytics workloads.
Cloud SQL: Cost-effective for workloads without horizontal scaling or global availability.
Cloud Spanner: For applications needing horizontal scaling or global availability.
Cloud spanner
Cloud Spanner Overview:
Designed specifically for the cloud to combine relational database features (schema, SQL, strong consistency) with non-relational horizontal scalability.
Supports petabyte-scale capacity and global transactional consistency.
Offers automatic synchronous replication for high availability and fault tolerance.
Key Use Cases:
Mission-critical systems: Financial services (e.g., transactions) and retail (e.g., inventory management).
Ideal for workloads requiring high throughput, global data consistency, and strong uptime guarantees.
Features:
Relational Features: Schema, SQL support, strong consistency.
Non-relational Features: Horizontal scalability, high availability, configurable replication.
Combines the best of both worlds for consistent, scalable systems.
Architecture:
Data is replicated across multiple Google Cloud zones using Google’s global fiber network.
Data placement is configurable to specific regions for global or regional availability.
Uses atomic clocks to ensure accurate and consistent updates.
Service-Level Agreements (SLAs):
Uptime varies between multi-regional and regional instances (check documentation for exact SLAs).
Decision Tree:
Choose Cloud Spanner if you need:
Horizontal scalability beyond relational databases.
Transactional consistency at a global scale.
Consolidation of multiple databases into one system.
High throughput and global availability.
If these features aren’t required and relational capabilities aren’t necessary, consider a NoSQL service like Cloud Firestore.
Migration:
Existing MySQL solutions can be migrated to Cloud Spanner using guides available in Google Cloud’s documentation.
AlloyDB
AlloyDB for PostgreSQL Overview:
A fully managed PostgreSQL-compatible database service designed for hybrid transactional and analytical workloads.
Combines a Google-built database engine with a multi-node cloud-based architecture to offer enterprise-grade performance, reliability, and availability.
Key Features:
Automated Administrative Tasks: Handles backups, replication, patching, and capacity management.
Adaptive Algorithms: Uses machine learning for tasks like vacuum management, storage and memory management, data tiering, and analytics acceleration.
Transactional Processing: 4x faster than standard PostgreSQL for transactional workloads.
Analytical Processing: Up to 100x faster than standard PostgreSQL for analytical queries.
Suitability:
Designed for enterprise workloads requiring high transaction throughput, large data sizes, or multiple read replicas.
Ideal for workloads that need real-time business insights.
High Availability:
99.99% uptime SLA, including maintenance periods, ensuring reliability.
Integration with Vertex AI:
Built-in support for machine learning via Vertex AI, allowing you to call ML models directly from the database for enhanced insights and analytics.
Firestore
Cloud Firestore Overview:
A fast, fully managed, serverless NoSQL document database designed for global-scale mobile, web, and IoT applications.
Offers live synchronization and offline support through client libraries.
Fully integrated with Firebase and Google Cloud Platform (GCP), enabling serverless app development.
Key Features:
ACID Transactions: Ensures transaction consistency—if any operation fails, the entire transaction fails.
Multi-region Replication: Offers strong consistency with automatic replication to ensure data safety and availability during disasters.
Sophisticated Queries: Enables complex queries on NoSQL data without performance degradation.
Flexible Data Structuring: Allows flexible data structuring, ideal for rapidly changing schemas.
Cloud Firestore vs Cloud Datastore:
Cloud Firestore is the next generation of Cloud Datastore.
In Datastore mode, it remains compatible with Cloud Datastore, improving the storage layer while maintaining Datastore's system behavior.
Native Mode introduces features like real-time updates, strong consistency, and a collection/document data model.
Use Cases:
Datastore Mode: Use for new server projects that need backward compatibility with Cloud Datastore.
Native Mode: Ideal for new mobile and web apps to take advantage of real-time updates and strong consistency.
Guideline:
Use Cloud Firestore in Datastore mode for server applications and native mode for mobile and web apps.
Existing Cloud Datastore users will be automatically upgraded to Cloud Firestore in the future.
Decision Tree:
If you need a highly scalable, adaptable database with low maintenance overhead, and can scale up to terabytes, Cloud Firestore is recommended.
If transactional consistency is not required, consider using Cloud Bigtable.
Cloud bigtable
Cloud Bigtable Overview:
A fully managed NoSQL database designed for petabyte-scale storage with low latency and high throughput.
Powers core Google services like Search, Analytics, Maps, and Gmail.
Ideal for both operational and analytical applications, including IoT, user analytics, financial data analysis, and machine learning.
Key Features:
High Throughput: Supports high read and write throughput with low latency, making it suitable for applications with high data volume.
HBase API Compatibility: Supports the open-source HBase API, enabling easy integration with existing systems.
Flexible Data Model: Uses sorted key/value maps, storing data in rows (representing entities) and columns (containing entity values).
Column Families and Qualifiers: Columns grouped by column families with unique column qualifiers, optimizing data storage.
Versioning: Stores multiple versions of data, allowing you to track changes over time.
Performance and Scalability:
Linear Scalability: Performance scales linearly with the addition of nodes, supporting up to hundreds of nodes for high throughput.
Sparse Tables: Cloud Bigtable tables are sparse—empty cells do not take up space, optimizing storage.
Automatic Adjustment: The system learns and adjusts to specific access patterns, distributing workload evenly across nodes.
Architecture:
Sharding: Data is divided into tablets (blocks of rows) for balanced workload distribution.
Colossus File System: Stores data on Colossus, Google's distributed file system, in SSTable format (ordered, immutable key-value maps).
Use Cases:
High-volume writes and real-time analytics (e.g., for IoT or machine learning).
Low-latency reads and writes (under 10ms), particularly for applications requiring strong consistency.
Cloud Bigtable vs Firestore:
Use Cloud Bigtable if you need to store more than 1TB of structured data, have high write volumes, need less than 10ms latency, or need HBase API compatibility.
If you don't need these features and need a service that scales down well, consider Cloud Firestore instead.
Cluster and Pricing:
The smallest Cloud Bigtable cluster requires 3 nodes, capable of handling 30,000 operations per second.
Charges are based on the number of operational nodes, regardless of usage.
Memorystore
Memorystore for Redis Overview:
A fully managed in-memory data store built on scalable, secure, and highly available infrastructure managed by Google.
Provides extreme performance for applications running on Google Cloud without the burden of managing complex Redis deployments.
Key Features:
High Availability: Instances are replicated across two zones with a 99.9% availability SLA.
Automated Management: Tasks like high availability, failover, patching, and monitoring are automated.
Scalable Performance: Supports instances up to 300 GB with 12 Gbps network throughput for high-performance applications.
Low Latency: Achieves sub-millisecond latency and high throughput.
Compatibility and Migration:
Fully compatible with Redis Protocol, enabling easy migration from open-source Redis to Memorystore with no code changes through the import/export feature.
Existing tools and client libraries work without requiring additional learning.
Scalability:
Start with the lowest tier and scale effortlessly with minimal impact on application availability.
Resource management
Resource manager
Resource Manager Overview:
Hierarchical Management: Resources are organized by project, folder, and organization, with policies set at each level.
IAM Policies and Inheritance:
Policies contain roles and members and are set on resources.
Inheritance: Resources inherit policies from their parent (e.g., organization, folder).
IAM Allow vs. Deny Policies: Allow policies are inherited, while deny policies override allow policies, restricting certain permissions for specific principals.
Billing and Resource Consumption:
Billing: Accumulates bottom-up, from projects to organization.
Resource Consumption: Measured in various units (rate of use, time, items, features).
Project-Level Billing: Each project is associated with one billing account, which tracks resource consumption.
Projects:
Role and Permissions: Projects are used for managing resources, billing, permissions, and APIs.
Identification: Projects are identified by:
Project Name (human-readable, not used by APIs),
Project Number (auto-generated by GCP),
Project ID (unique ID derived from the project name).
These details are available in the GCP Console or via the Resource Manager API.
Resource Hierarchy:
Types of Resources:
Global: Images, snapshots, networks.
Regional: External IP addresses.
Zonal: Instances, disks.
Resources are always tied to a project for billing and reporting purposes.
Quotas
Quotas Overview:
Project Quotas: Every resource in Google Cloud is subject to quotas or limits within a project. These quotas help manage resource allocation and prevent overconsumption.
Types of Quotas:
Resource Creation Quotas: Limits on how many resources can be created per project (e.g., 15 VPC networks per project).
API Rate Limits: Restrictions on how quickly API requests can be made (e.g., 5 administrative actions per second for the Cloud Spanner API).
Regional Quotas: Limits specific to regions (e.g., 24 CPUs per region by default).
Quota Adjustments:
As your usage grows, you can request quota increases through the Quotas page in the Cloud Console.
This allows proactive adjustments to accommodate future needs.
Purpose of Quotas:
Prevent Runaway Consumption: Quotas protect against errors or malicious attacks that could lead to excessive resource usage (e.g., accidentally creating 100 instances instead of 10).
Prevent Billing Surprises: Quotas help control costs and avoid unexpected spikes in billing.
Sizing Considerations and Review: Quotas encourage reviewing resource needs to avoid over-provisioning.
Resource Availability: Quotas are the maximum allowed, but do not guarantee resource availability. For instance, even if you have quota for local SSDs, you cannot create them if the region runs out of available SSDs.
Quota vs. Availability:
Quotas limit how much of a resource you can create, but they don’t guarantee availability of the resource.
For example, a region may be out of local SSDs even if your quota hasn’t been reached.
Labels
Labels in Google Cloud:
Purpose: Labels provide granular organization for Google Cloud resources beyond what projects and folders can offer. They are key-value pairs that help categorize resources.
Managing Labels:
You can create and manage labels using the GCP Console, gcloud, or the Resource Manager API.
Each resource can have up to 64 labels.
Use Cases:
Environment: Label resources to define the environment (e.g.,
environment:production,environment:test), making it easier to search and list resources by environment.Team or Cost Center: Use labels like
team:marketingorteam:researchfor cost accounting and budgeting.Component Identification: Label resources by their component type (e.g.,
component:frontend,component:redis).Owner or Contact: Add labels to define an owner (e.g.,
owner:gaurav) or a primary contact (e.g.,contact:opm).State of Resource: Labels like
state:inuseorstate:readyfordeletionhelp define the current state of resources.
Labels vs. Network Tags:
Labels are user-defined key-value pairs that organize resources and can be used for billing and reporting.
Network Tags are also user-defined, but they are applied to instances and mainly used for networking purposes, such as applying firewall rules or custom static routes.
Billing
Budget Setup:
You can create a budget for your project to track spend.
The budget can be set based on a specific amount or the previous month's spend.
Alerts can be configured to notify you via email when spending exceeds specific percentages (e.g., 50%, 90%, 100%).
You can also set alerts for forecasted overspending by the end of the budget period.
Notifications:
Alerts are sent to billing admins via email when thresholds are exceeded.
Pub/Sub notifications can also be used to programmatically receive spend updates. You could create a Cloud Function to automate cost management based on these updates.
Using Labels to Optimize Spend:
Labels can help you track resources and manage costs. For example, labeling VM instances by region helps you identify potential high-cost areas, like instances sending traffic across continents.
In such cases, relocating instances or using services like Cloud CDN (for caching content closer to users) can help reduce networking costs.
Export Billing Data:
Exporting your billing data to BigQuery allows you to analyze spend in a scalable, SQL-based platform.
You can create queries to assess costs and even visualize spending trends with Looker Studio. This tool helps turn your data into customizable dashboards and reports.
Optimizing Spend with Queries and Reports:
BigQuery lets you query your billing data, and Looker Studio allows you to create interactive reports, which you can slice and dice using your labels to understand cost distribution.
Resource monitoring
Operation suite
Integrated Monitoring, Logging, Diagnostics:
Google Cloud’s operations suite provides integrated services for monitoring, logging, error reporting, and fault tracing.
These services are tightly integrated to offer core visibility into the cloud platform quickly, with smart defaults that enable rapid discovery of resources.
Platform Management:
It enables management across platforms, including both Google Cloud and AWS.
The suite dynamically discovers cloud resources and application services in Google Cloud, helping to streamline the monitoring process without additional configuration.
Open-Source Integrations and Agents:
The suite supports open-source agents and provides numerous integrations to ensure smooth operation and monitoring across different environments.
Powerful Data and Analytics Tools:
It grants access to advanced data and analytics tools to gain deeper insights into your cloud operations.
With collaboration with third-party software, it enhances the flexibility and extensibility of your cloud management solutions.
Pricing and Flexibility:
You pay only for what you use, and there are free usage allotments, which makes it easier to get started without any upfront fees or commitments.
Advantages Over Traditional Software:
Unlike other environments where monitoring and diagnostic services are managed by separate tools, Google Cloud’s operations suite integrates these functions into a single comprehensive service.
This integration is crucial for creating reliable, stable, and maintainable applications.
Monitoring
Cloud Monitoring Overview:
Part of Google Cloud’s operations suite; enables monitoring, insights, and alerts for platform, system, and application metrics.
Supports monitoring via dashboards, charts, and alerts, such as uptime and health checks.
Key Components:
Metrics Scope:
Root entity for monitoring configurations.
Can include 1–375 monitored projects, with one scoping project defining the scope name.
Centralizes monitoring but retains metric data and logs within individual projects.
Custom Dashboards:
Displays metrics like CPU utilization, network traffic, and dropped packets.
Supports filters, groups, and aggregation for clarity.
Alerting Policies:
Notify when specific conditions (e.g., high network egress) occur.
Alerts sent via email, SMS, or other channels.
Best Practices:
Alert on symptoms (e.g., failing queries), not causes.
Use multiple notification channels to avoid single points of failure.
Customize alerts for specific audiences and actions.
Avoid excessive, noisy alerts.
Uptime Checks:
Tests service availability via HTTP, HTTPS, or TCP from global locations.
Configurable for App Engine apps, Compute Engine instances, URLs, AWS instances, etc.
Failures occur if responses are not received within the timeout period.
Ops Agent:
Primary tool for collecting telemetry data from Compute Engine instances.
Provides internal metrics (e.g., memory usage) beyond hypervisor-level data.
Supports major OSes (CentOS, Ubuntu, Windows) and third-party application monitoring.
Custom Metrics:
Enable tracking of unique application-specific metrics (e.g., number of game server users).
Useful for scaling events or tailored monitoring requirements.
Autoscaling with Metrics:
Maintains metrics at a target value by adjusting VM instances in managed instance groups.
Filters can be applied for precise autoscaling based on individual metric values.
Documentation:
Extensive guides for dashboards, metrics, alerts, uptime checks, and custom configurations available for detailed insights.
Logging
Cloud Logging Overview:
Fully managed service for storing, searching, analyzing, monitoring, and alerting on logs from Google Cloud and AWS.
Can ingest application and system log data from thousands of VMs.
Includes a Logs Explorer UI and an API for programmatic log management.
Log Retention and Exporting:
Logs are retained for 30 days but can be exported to Cloud Storage, BigQuery, and Pub/Sub for longer-term use.
Cloud Storage: Best for storing logs longer than 30 days.
BigQuery: Allows for fast analysis of logs using SQL queries, handling gigabytes to petabytes of data.
Example: Analyze network traffic to optimize capacity or expenses.
Can visualize logs in Looker Studio for easy-to-understand reports and dashboards.
Pub/Sub: Streams logs to applications or endpoints for real-time processing.
Log Analysis Use Cases:
Query logs to understand traffic growth, forecast capacity, and optimize network traffic expenses.
Example: Identifying top IP addresses exchanging traffic with a server to optimize infrastructure or block unwanted access.
Error reporting
Error Reporting Overview:
Counts, analyzes, and aggregates errors in cloud services.
Centralized Interface:
Provides a centralized error management interface with sorting and filtering capabilities.
Enables real-time notifications when new errors are detected.
Supported Services:
Available for App Engine (both standard and flexible), Apps Script, Compute Engine, Cloud Functions, Cloud Run, Google Kubernetes Engine, and Amazon EC2.
Supported Languages:
Can process exception stack traces for Go, Java, .NET, Node.js, PHP, Python, and Ruby.
Tracing
Cloud Trace Overview:
A distributed tracing system for collecting latency data from applications and displaying it in the Google Cloud console.
Tracks how requests propagate through the application and provides near real-time performance insights.
Automatic Latency Analysis:
Automatically analyzes application traces to generate detailed latency reports.
Identifies performance degradations and surfaces issues for improvement.
Integration:
Captures traces from App Engine, HTTP(S) load balancers, and applications instrumented with the Cloud Trace API.
Application Performance Management:
Focuses on managing the time it takes for applications to handle incoming requests and operations, crucial for overall performance.
Foundational Tools:
Built on tools used at Google to manage high-scale services, ensuring reliability and efficiency.
Profiling
Cloud Profiler Overview:
Continuously analyzes CPU or memory-intensive functions in applications to identify performance issues.
Production vs. Development:
Development environment measurements don’t always reflect production performance.
Production profiling techniques often slow down code execution or only inspect a small portion of the code.
Low-Impact Profiling:
Profiler uses statistical techniques and low-impact instrumentation to analyze performance across all production application instances without slowing down execution.
Cross-Environment Support:
Supports applications running on Google Cloud, other cloud platforms, or on-premises.
Compatible with languages like Java, Go, Node.js, and Python.
Partner integrations
Technology Partners Ecosystem:
Google Cloud supports a growing ecosystem of technology partners to expand IT operations, security, and compliance capabilities.
BindPlane Integration:
BindPlane (from Blue Medora) helps collect metrics and logs, pushing them into Google Cloud's core observability platform.
Once logs are ingested into Cloud Logging, they can be viewed, searched, and used to create metrics like any logs from Google Cloud.
Real-time Log Viewing & Alerts:
Logs can be viewed in real time in the Google Cloud console, and log-based metrics can be used to set up alerts.
Log Data Export to Splunk:
Pub/Sub temporarily stores log messages before delivering them to Splunk.
Dataflow pipelines extract logs from Pub/Sub and deliver them to Splunk.
A secondary Dataflow pipeline ensures logs are resent if delivery fails.
Splunk Analysis:
Logs can be analyzed in Splunk (on-premises, in Google Cloud, or hybrid approach).
Dataflow template used to export data from Pub/Sub to Splunk for detailed analysis.