Basic Topics (Covered on Day 1)

• PCIe Architecture Overview
  • Introduction
  • Basic operation
  • Transaction types
  • Layers and responsibilities overview
• NVMe Introduction
  • NVMe concepts and definitions
  • Comparison with SATA
  • NVMe Host-Controller Interface overview
  • Queue Management
    • Doorbell Head and Tail registers
    • Execution sequence
    • Tracking completion status
• Commands
  • NVMe Admin Commands
  • General Structure of commands
  • Identify command options and results
  • Creation and management of Queues
  • Asynchronous Event Notification
  • NVMe IO Commands
    • Read and Write command structure
    • Optimizing write commands
    • Avoiding write amplification
    • Buffer addressing modes (PRPs vs. SGLs)
    • Priority and Arbitration of commands
    • Completing commands
    • Handling Metadata (DIF/DIX)
    • Get Features / Set Features – full coverage of features

Additional Topics (Covered on Day 2)

• Architecture
  • Error reporting, Error Reporting Structures
    • Asynchronous Events
    • Completion status
    • Log pages
  • Other log pages
  • Firmware updates
  • Controller registers
  • Controller initialization
  • Power Management
  • Reservations
•  Other NVMe commands
  • Write uncorrectable
  • Write zeroes
  • Compare
  • Dataset management
  • Configuration overview
• Summary of changes for 1.3
  • Boot partitions
  • Thermal management by host
  • Write streams
  • Deallocation
  • Virtualization support
  • Debug support – Telemetry, initiated by host or device
  • Self-test options
  • Sanitize command
  • New commands
• Summary of changes for 1.4
  • NVM Sets
  • I/O Determinism
  • Read recovery levels
  • Persistent Memory Region (PMR)
  • Asymmetric Namespace Access (ANA)
  • Namespace write protect
  • Persistent event log
  • Rebuild assist
  • Endurance groups
  • I/O Performance and Endurance Hints
  • Traffic-based Keep Alive
  • Vendor-specific UUIDs
  • Verify - new command
  • Administrative controller
  • SQ Associations

Additional Topics (Covered on Day 3)

• Summary of changes for 2.0 through 2.2
  • Flexible Data Placement
  • Live Migration
  • Dispersed Namespaces (NS)
  • Device Personalities
  • Rotational (Spinning) media support
  • Telemetry Log Changes
  • Persistent Event Log Enhancements
  • Updated I/O Commands
    • NVM Command Set
    • Computational Command Set Overview
  • Updated Admin Commands
  • Controller Register Enhancements
    • CMB Write Buffer registers
    • Time To Ready
    • Shutdown options
  • Zoned Namespaces Overview
  • Key Value Overview

NVMe-oF Topics (Covered on Day 4)

• Introduction
  • Motivation for moving NVMe to fabrics
    • Improve storage network performance
    • Avoid limitations of PCIe, extend NVMe protocol to the fabric
    • Optimize access latency across the network
  • Differences from base NVMe
  • Initialization Background
• Fabric Discovery
  • Discovery Service, Discovery Controllers
  • Discovery Mechanism
• NVMe-oF Initialization
  • Fabrics Commands
  • Device Setup
  • Transport mapping/binding RDMA
  • Introduction to RDMA (Remote Direct Memory Access)
  • Protocol Translation
  • RDMA Examples
• RDMA Concepts
  • Motivation: Kernel bypass – reduce CPU involvement
  • Zero-copy operation when possible
  • RNIC (RDMA-aware Network Interface Card)
  • Pinned remote memory and storage
  • Command-based Flow Control
  • Read and Write Command Examples
• NVMe-oF Details
  • Controller Initialization
  • Connections and Associations
  • Static vs. Dynamic Controller model
  • Properties instead of registers
  • Authentication
  • Encapsulation
  • Accessing RDMA Queue pairs
  • Data transfer: Command, Response, Data, Flow Control
  • Fabrics Commands and Responses
• RNIC Details
  • Software Layers
  • RDMA Verbs
  • Memory Regions and Windows
• Ethernet overview
  • TCP/IP: addressing, layers, packets
  • RDMA models added to it: RoCE and iWARP
    • Overview of iWARP
      • Added to TCP/IP, so packets are routable and don’t need DCB
      • Messages
      • Layers
      • Advantages and disadvantages
    • Overview of RoCE
      • Version 2 packets include routing information, version 1 did not
      • Use of Data Center Bridging (DCB) to manage routing
      • Flow control
      • Soft RoCE
      • Advantages and disadvantages