Fiber Channel Primer

Lately I’ve been spending a lot of time in a software product called VirtualWisdom (By VirtualInstruments).  This software dives into our fiber channel network down to the frame level and reports tons of information about performance and the status of our HBAs and Fiber Channel links.  It’s a ton of data, and I’ve spent the better part of a week diving into Fiber Channel basics to try and understand the data better.  Below are some notes I’ve put together from a number of sources (listed below).  There’s a lot out there, but it’s hard to get it all in one place in an easy-to-understand format. I’m hoping this helps someone out someday.  I’m pretty sure I have all of the data accurate, but please don’t hesitate to contact me with any corrections.

In this first post, I’m just going over the basics.  In subsequent posts, I want to dive into error detection and handling.  This is one of the key areas VirtualWisdom gives you great insight into, and understanding the different error types and where they come from will help you make sense of  the data.

Fiber Channel Topology

Three basic types:

  • Fabric (Switched)
  • Point-to-point
  • Arbitrated Loop

FC Toplology

Port Types

  • F-Port is a fabric port.  This port is on the switch-side.
  • N-Port is a node port.  This is on the initiator or target side.
  • NL-Port is a node loop port.  This refers to a node whe there are multiple nodes in an arbitrated loop.
  • FL-port is a fabric loop Port.  Switch side of the of an NL-port connection.
  • Fx-port refers to either a F-port or an FL port.
  • Nx-port refers to N-port or an NL-port
  • E-Port refers to a port on either side of a switch ISL (Inter-switch link)

FC Layers

FC Layers 1 FC Layers 2

FC-0 – The Physical Layer

Defines the physical link, including the fiber, connectors and optical and electrical parameters for data rates.  This layer is responsible for actually sending the bits of data across the link.

FC-1 – The Encode/Decode Layer

This layer takes character data and encodes it into link pulses.  It encodes the data 8-bytes at a time into 10-byte transmission characters.  This layer prepares the streams of bits for the FC0 layer to send.

FC-2 – The Framing Layer

Serves as the transport mechanism of FC.  Orders the data into frames, and defines the service class to be used.  Manages the sequence of data transfers.

Responsible for breaking down data into frames.

Building blocks:

  • Ordered set
  • Frame
  • Sequence
  • Exchange
  • Protocol

Ordered Set

4-byte transmission words

  • Contains data and special characters that have specific meaning
  • Three major types of Ordered Sets defined by signal protocol
    • Frame Delimiters
      • Start-of-Frame (SOF)
      • End-of-Frame (EOF)
    • Primitive Signals
      • Receiver Ready (R_RDY)
        • Indicates a port buffer can handle more frames
      • Idle
        • Transmitted to indicate a port is ready for Tx and Rx
    • Primitive Sequence
      • Transmitted continuously to indicate a certain status or condition of a port
      • Will be recognized when 3 consecutive sets of the same Primitive Sequence is received
      • Offline (OLS), Not operational (NOS), Link Reset (LR) and Link Reset Repsonse (LRR) are supported Primitive sequences

Frame

Frame

  • Frames are basic building blocks of an FC connection
  • Contains information:
    • Data to be transmitted (payload)
    • Source and destination ports
    • Link control information
  • Two basic types
    • Data Frames
      • Link_Data Frames
      • Device_Data Frames
    • Link Control Frames
      • Acknowledge (ACK)
      • Link_Response (Busy, Reject)

Sequence

  •  Is a set of one or more frames transmitted unidirectionally from one N-Port to another N-Port
  • Each frame in the sequence has a sequence count number
  • Error recovery is usually performed at the boundaries of sequences

Conversation

Exchange

  •  Is one or more nonconcurrent exchanges in a single operation.
  • Can be bidirectional
  • Think of it as a “Conversation”
  • In one exchange, only one sequence can be active at a time
  • Sequences of multiple exchanges can be concurrently active

Flow Control

  •  Depends on the class of service
  • Class 1 uses end-to-end flow control
    • Credits tracked only at the start and end points of the transmission, not the hops in between
  • Class 3 uses buffer-to-buffer
    • Credits tracked at each point of communication between the source and destination
    • Class 2 uses both
  • Managed by Sequence Initiator (Source) and Sequence Recipient (Destination) ports using two counters:
    • Credit
      • Number of credits issued by the recipient to the transmitter
    • Credit_CNT
      • Number of frames that have not been acknowledged by a R_RDY from the recipient
  • If there are not buffer credits available for incoming data, a “Busy” frame is sent back to the transmitter
  • If a frame error is received, a “Reject” frame is sent back

Service Classes

  • Class 1 is point-to-point
    • Sends and ACK for every frame sent
    • Frame delivery is in-order
    • Locks the entire link between the sender and trasmitter, does not share
    • Good for sustained, high-bandwith operations
  • Class 2 is frame-switched and multiplexed
    • Sends ACK for every frame sent
    • Frame delivery is not guaranteed
    • Connectionless
    • Good when the time needed to setup connection is longer than it takes to send the message
  • Class 3 is identical to class 2, but no ACK is sent on frame delivery
    • Ideal for real-time communication.  Data received too late is not of value anymore.
    • Frame delivery is not guaranteed

FC-3 – Common Services Layer

  • Striping – allows you to multiply bandwith by striping data over multiple N_Ports in parallel
  • Hunt Groups – Allows more than one N_Port to respond to a request using an alias address.   Reduces chance for a busy port.
  • Multicast – Allows broadcast of single signal to multiple destination ports.

FC-4 – Application Layer

  • Defines the interfaces that can transmit over FC
  • Supports both network and channel protocols
  • Suports
    • SCSI
    • IPI – Intelligent Peripheral Interface
    • HIPPI – High Performance Parallel Interface
    • IP
    • AAL5
    • FC-LE – Link Encapsulation
    • SBCCS
    • IEEE 802.2

References

http://www.virtualinstruments.com/sites/default/files/documents/wp-top-10-brocade-problems_0.pdf

http://jeff.nieusma.com/docs/fibre_channel.html

http://hsi.web.cern.ch/HSI/fcs/spec/overview.htm

http://www.rfc-editor.org/rfc/rfc4983.txt

http://www.t11.org/ftp/t11/pub/fc/fs-4/13-113v0.pdf

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s