Purity OS, Data Reduction & Snapshots

Purity OS Features

• Purity OS is installed on both controllers of the FlashArray.

• Features:

  • Thin Provisioning: No space is reserved upfront when creating volumes or file systems.

    • Space is allocated on-demand as the host writes data, ensuring no unnecessary blocking of space.

  • Data Reduction: Deduplication and compression are applied to every volume and file system.

Deduplication

• Process of identifying and removing duplicate data.

• If the incoming data already exists in the system, the index table points to the existing data instead of storing it again, even if it's owned by another volume.

• Example: WhatsApp messages.

  • Multiple users (volumes) sending the same message ("Good morning") point to a single instance of that message on the WhatsApp storage.

• Ensures no space wastage and significant space savings.

Compression

• Applies to unique data.

• Removes sub-blocks of zeros and known patterns.

• Example: "Why Purity?" (10 bytes) can be compressed by removing the space (9 bytes).

• Known Pattern Removal:

  • If the data has repeating patterns (e.g., AABBCCCDDDEEEFFF), it can be shortened (ABCDEFX3).

Compression Example

• AABBBCCCDDDEFF (18 bytes) can be reduced to ABCDEFX3 (8 bytes).

• The processor decompresses the data when the host sends a read request.

Inline vs. Deep Compression

• Other storage systems:

  • Perform inline deduplication and compression.

  • Data is deduplicated and compressed before writing to the disk.

• Pure Storage FlashArray:

  • Does deduplication and minimal compression in-line.

  • Offloads deep compression to a dedicated FPGA card (Deep Compression Algorithm or DCA).

Why Use DCA?

• CPU can focus on other IO operations, avoiding performance degradation.

• CPU does deduplication and minimal compression, offloads the deep compression to DCA.

• DCA further compresses the data before writing to the disk.

• Deep compression happens in two levels, resulting in better data reduction efficiency.

• After writing to the disk, Purity OS checks for additional data reduction opportunities at the block level (post deduplication).

Data Flow Example

• Host 1 & Host 2: Connected to FlashArray via SAN (Fiber Channel or iSCSI).

• FlashArray: Contains two controllers and DFMs (Direct Flash Modules).

• Volume Creation: Creating a volume creates an index table (e.g., V1 - 50GB, V2 - 80GB).

• Host Writes Data:

  • Host 1 writes data "ABC".

  • Data is kept in the cache (DRAM) of both controllers and staged in NVRAM, and the host is acknowledged.

• Deduplication Process:

  • CPU generates a hash value for "ABC" (e.g., H420).

  • The system checks the system index table to see if that value exists.

  • If the value is new, it's added to the index table and written to the disk. The LBA is the hash value itself.

• Host 2 Writes Data:

  • Host 2 writes data "ABX".

  • A hash value (e.g., H430) is generated, and it is checked against the system index table.

  • If the value is new, it's added to the index table and written to the disk.

• Post Deduplication

  • After data is written to disk, Purity OS compares at a 512-byte block level to identify any further duplication. If duplicates exist:

    • One copy is removed and pointers are redirected.

FlashArray Data Reduction and Efficiency

• FlashArray performs both inline and post deduplication, leading to better data reduction ratios compared to other storage systems.

RAID Achieve

• Uses RAID-6 for data protection.

• Purity OS is installed on both controllers.

• FlashArray works on an active-standby architecture.

• The secondary controller monitors the primary controller and takes over instantaneously in case of a failure.

Non-Disruptive Upgrade (NDU)

• Purity OS upgrades can be performed without any downtime from the host perspective.

• One controller reboots while the other takes over.

Purity OS Features

• Protect: Supports snapshots, clones, and three modes of replication.

• Always-On Encryption: Data-at-rest encryption.

  • Data is spread across multiple disks, and the encryption key is also spread across multiple disks.

  • If a drive is stolen, the data cannot be decrypted.

• Optimize: An optional feature.

• Quality of Service (QoS): Limits the number of IOs per second to prevent resource exhaustion for a particular volume, thus ensuring resources are available for other volumes.

• Variable Block Engine: Purity OS takes IOs from different hosts with varying sizes, splits them into equal sizes, processes them, puts them in sequential stripes, and stores them.

• Flash Reliability: Purity OS reduces the amount of data written to the disk due to deduplication and compression.

  • Reduces wear and tear on flash drives, extending their lifespan.

  • Analogy:

    • Avoid unnecessary trips (writes) to reduce fuel consumption and wear and tear on a car.

• NVMe Based SSDs: No flash translation layer, the processor directly communicates with NAND gates.

• Global Flash Management and Garbage Collection: Reclaim space when data is deleted.

  • Garbage collector reclaims the space and puts the fragmented data back to sequential order.

Pure One Meta

• AI platform that proactively analyzes logs, identifies problems, and provides recommendations.

Pure CloudSnap

• Feature that allows offloading snapshot data to block storage or S3 bucket on FlashBlade.

• Frees up space on the array without affecting the base volume and its attached snapshot.

• Snapshot data is needed to roll back the data for the base volume. Usually, snapshots are not needed on a daily basis.

Purity OS Updates

• Pure Storage releases new versions of Purity OS with new features.

• Review the release notes before upgrading to understand new features and addressed issues.

• Schedule upgrades through Pure One for support team assistance.

  • The support team checks logs, does health checks, and assists during the upgrade process.

  • Faster resolution time in case of any issues.

IO Handling

• FlashArray models have two controllers.

• Both controllers present paths to the host as active-optimized paths.

• Host operating systems use MPIO with a round-robin policy, allowing IO to both controllers.

• Backend: One controller is primary, the other is secondary.

• Only one controller processes data and writes to the disk.

• Performance is guaranteed even if one controller fails.

Front-End Disks

• The minimum number of disks is 10, and the maximum is 28 for almost all FlashArray models except for FlashArray//XL.

• For FlashArray//XL, the maximum number of disks is 20.

• All disks in a base enclosure are of the same type (DFM - Direct Flash Module).

• They become part of a single write group automatically.

Write Groups

• Write groups are created automatically by Purity OS.

• Purity OS determines the number of drives and their capacity.

• One or two write groups might be created based on whether all drives have equal capacity.

• If there are 10 drives of, say, 2.2 TB each, and then another 10 drives of, say, 4 TB each, two write groups will be created: One for the first 10 drives, and another for the other 10 drives.

• When a volume is created, RAID segments are created from the write group and assigned to the volume's index table.

Data Handling

• Host sends data, which is kept in the DRAM of both controllers, and the host is acknowledged.

• Controllers do deduplication and compression before writing the data.

• To deduplicate, a hash value is generated, it is checked against the volume lookup table, and data is compared.

• If the hash value exists, the data is compared to confirm it's the same.

• If there is a match, the data is not written. Instead, the index table is updated.

• If unique, the data is compressed and written to the disk.

• Post-deduplication is performed after writing the data to the disk, and compares data at the 512-byte block level.

Data Reduction Efficiency

• Achieved through:

  • Inline deduplication.

  • Post deduplication.

  • Compression.

  • Deep compression.

  • Pattern removal.

Write Groups

• One write group contains a maximum of 28 drives.

• If the base enclosure has 48 drives (XL series), two write groups are created even if all drives are of the same type.

  • One write group contains 1 eight point drives.

  • The other write group contains the remaining drives.

• When multiple write groups exist, the volume can grow and create RAID sets from all write groups.

  • It does not choose which write group to use; all are used.

Data Storage and Garbage Collection

• Data is written in order, without leaving space in between.

• As the host deletes data randomly, there is fragmentation.

• Garbage collector reclaims the space, puts it into a sequential stripe, and writes the data back.

Pure Test Drive Lab Environment

• The presenter will be using the Pure Test Drive to demonstrate basic provisioning.

• The lab includes FlashArray and FlashBlade labs.

• The topology includes two FlashArrays, one Windows host, one Linux host, and one VMware cluster.

• The presenter will be using the Windows host as the management host.

Partner Portal Access

• Access to the test drive is through the partner portal.

• The link will be provided, and it is up to the user to access it.

• No lab demonstrations are accessible through the presenters account because they are not a Pure Storage partner.

• Login to the partner portal, then click on Training and Tools and then click on Tools.

• Scroll down to Pure Test and click on it.

GUI Access

• Open a browser and enter management IP address or URL (flasharray1.desktop.local).

• Default username: pureuser.

• Default password: pureuser.

• Dashboard shows high-level overview of capacity utilization and overall array performance.

Creating a Volume

• Go to Storage > Volumes.

• Click on the plus sign.

• Enter the volume name (e.g., Volume0).

• Specify the size (e.g., 50GB).

• QoS: Optional.

• Click Create.

• Thin provisioning and data compression are enabled by default.

Exporting a Volume

• Create a host contact for the host in question by entering the IQN or WWPN of that host.

• Otherwise, the host will not be able to connect to the flash array.

• Then click on the volume that was created, connect the host and click connect.

• The flash array admin should be able to see the volume listed via disk management after connecting the host.

File Systems and Volumes

• Volumes require the host operating system to determine the file system for raw storage.

• File systems are already formatted, so the host operating system can connect immediately and write data.

Creating a File System

• Lab environment: FlashArray is connected to the local area network and assigned an IP address.

• File systems can be broadcasted over LAN.

• Assign SMB for Windows, NFS for Linux, or both for simultaneous access.

Connecting with Active Directory

• Go to Settings > Access > Active Directory Accounts.

• Click on the plus sign.

• Enter the details of the array, domain name, and credentials.

• The FlashArray will appear in Active Directory Users and Computers.

Data IP

• Access any client by connecting to data IP address.

File System Creation

• Go to Storage > File System.

• Click on the plus sign.

• Set a name for the file system.

Volume vs. File System

• For volume (if just volume is exposed), the storage operating system will not put any file system.

• The exported host operating system will see it as a raw disk.

• Here, the purity OS will put a file system.

• Create a directory within the file system. Assign the policy to it.

• The same directory can be assigned to both NFS and SMB policy.

• The NFS and SMB share will appear as folder name Users.

• Users will then start storing the files.

Directory Exports

• Unlike volume, you do not have to specify to which host it is to be exported (exported to network).

• Anything in the network can access the directory.

• Windows can access it via: \ or \

Linux Host Access

• Connect to the file system:

  • showmount -e <FlashArrayName>

• First need to:

  • Mount file directory to linux

  • sudo mkdir <dirName>

  • sudo mount <directoryOnFlashArray> <linuxDirectoryName>

WhatsApp Demo

• Data will be shared among different users having different operating systems.

Multiple Cloud Benefit

Access to shared files can be made to multiple users with different operating systems.

Snapshot Basics

• Point in time copy of data

• Does not duplicate data unless changes are made

• Copy of index table

Use cases of taking snapshot

• Restore: Take it to restore the data

• Periodic: Replication software in order to differentiate the data

• Backup: Used by backup software as well

Why Snapshots

• Pointer based replica of data

• Don't consume a lot of space.

• Volume doesn't need to be offline

• Number of Sanpshots is infinite: Will not consume space

Snapshots

• Space is not completely empty:

• Making clone: It is another volume altogether. No dependency to source volume
  Make snapshot with the copy of the date, will copy entire time

• Initial snap will not copy space, only when original volume changes where space will be impacted.

• Replication: You mean like the snapshot, it will be like a photocopy. Will Correct, those size, and it replicated to another array. And after application, there was a changes happen on the data side. Correct. We consume the space on the source and replica replica destination side also or only the source side?
  It is on the destination side the data will consume because the replication happened

Deeper Explanation

• Take volume

• Set you volume 1, write some data which the letter gets copied into the cache. Once it does and is unique. Then goes directly straight into the index table. After one gb is used, the data is replicated. Then when snapshot copies with new idex table and pointers.

Use Cases of Snapshot Cont.

• Base volume has a, then snapshot will point to the A

• Snapshot, where space is not available due to exclusive copy

File Services

Like the backup application Veritas Commvault

Create Production Group, it is volumes inside a basket on a production group