Product SiteDocumentation Site

Red Hat Storage Software Appliance 3.2

User Guide

User Guide for Red Hat Storage Software Appliance

Edition 1

Red Hat Engineering Content Services

Red Hat Inc.

Legal Notice

Copyright © 2011 Red Hat Engineering Content Services.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
All other trademarks are the property of their respective owners.


1801 Varsity Drive
Raleigh, NC 27606-2072 USA
Phone: +1 919 754 3700
Phone: 888 733 4281
Fax: +1 919 754 3701

Abstract
This guide introduces Red Hat Storage Software Appliance, describes the minimum requirements, and provides step-by-step instructions to install the software in your environment.
Preface
1. Audience
2. License
3. Document Conventions
3.1. Typographic Conventions
3.2. Pull-quote Conventions
3.3. Notes and Warnings
4. Getting Help and Giving Feedback
4.1. Do You Need Help?
4.2. We Need Feedback!
1. Introducing Red Hat Storage Software Appliance
2. Preparing to Install Red Hat Storage Software Appliance
2.1. Red Hat Storage Software Appliance Installation Overview
2.2. Checking Red Hat Storage Software Appliance Minimum Requirements
3. Downloading and Preparing Installation Media
4. Installing and Updating Red Hat Storage Software Appliance
4.1. Installing From USB Stick
4.2. Installing from .ISO Image
4.3. Booting From PXE
4.4. Registering to Red Hat Network (RHN)
4.5. Setting up Software Updates
5. Migrating to Red Hat Storage Software Appliance
5.1. Migration Overview
5.2. Pre-Migration Task
5.3. Migrating your Storage Software Appliance
5.4. Validating your Migration
6. Starting and Stopping the glusterd Daemon
6.1. Starting and Stopping glusterd Manually
6.2. Starting glusterd Automatically
7. Gluster Console Manager – Command Line Utility for Configuration and Management
8. Trusted Storage Pools – Preparing GlusterFS for Management
8.1. Adding Servers to Trusted Storage Pool
8.2. Removing Server from the Trusted Storage Pool
9. Setting Up GlusterFS Server Volumes
9.1. Creating Distributed Volumes
9.2. Creating Replicated Volumes
9.3. Creating Striped Volumes
9.4. Creating Distributed Striped Volumes
9.5. Creating Distributed Replicated Volumes
9.6. Starting Volumes
9.7. Displaying Volume Information
10. Accessing Data - Setting Up GlusterFS Client
10.1. Gluster Native Client
10.1.1. Installing the Gluster Native Client
10.1.2. Mounting Volumes
10.2. NFS
10.2.1. Using NFS to Mount Volumes
10.3. CIFS
10.3.1. Using CIFS to Mount Volumes
11. Managing GlusterFS Volumes
11.1. Tuning Volume Options
11.2. Expanding Volumes
11.3. Shrinking Volumes
11.4. Migrating Volumes
11.5. Rebalancing Volumes
11.5.1. Rebalancing Volume to Fix Layout Changes
11.5.2. Rebalancing Volume to Migrate Existing Data
11.5.3. Rebalancing Volume to Fix Layout and Migrate Existing Data
11.6. Stopping Volumes
11.7. Deleting Volumes
11.8. Triggering Self-Heal on Replicate
12. Managing GlusterFS Geo-replication
12.1. Replicated Volumes vs Geo-replication
12.2. Preparing to Deploy GlusterFS Geo-replication
12.2.1. Exploring Geo-replication Deployment Scenarios
12.2.2. GlusterFS Geo-replication Deployment Overview
12.2.3. Checking Geo-replication Minimum Requirement
12.2.4. Setting Up the Environment for Geo-replication
12.3. Starting GlusterFS Geo-replication
12.3.1. Starting Geo-replication
12.3.2. Verifying Successful Deployment
12.3.3. Displaying Geo-replication Status Information
12.3.4. Configuring Geo-replication
12.3.5. Stopping Geo-replication
12.4. Restoring Data from the Slave
12.5. Best Practices
13. Monitoring your GlusterFS Workload
13.1. Running GlusterFS Volume Profile Command
13.1.1. Start Profiling
13.1.2. Displaying the I/0 Information
13.1.3. Stop Profiling
13.2. Running GlusterFS Volume TOP Command
13.2.1. Viewing Open fd Count and Maximum fd Count
13.2.2. Viewing Highest File Read Calls
13.2.3. Viewing Highest File Write Calls
13.2.4. Viewing Highest Open Calls on Directories
13.2.5. Viewing Highest Read Calls on Directory
13.2.6. Viewing List of Read Performance on each Brick
13.2.7. Viewing List of Write Performance on each Brick
14. Managing Directory Quota
14.1. Enabling Quota
14.2. Disabling Quota
14.3. Setting or Replacing Disk Limit
14.4. Displaying Disk Limit Information
14.5. Updating Memory Cache Size
14.6. Removing Disk Limit
15. POSIX Access Control Lists
15.1. Activating POSIX ACLs Support
15.1.1. Activating POSIX ACLs Support on Sever
15.1.2. Activating POSIX ACLs Support on Client
15.2. Setting POSIX ACLs
15.2.1. Setting Access ACLs
15.2.2. Setting Default ACLs
15.3. Retrieving POSIX ACLs
15.4. Removing POSIX ACLs
15.5. Samba and ACLs
15.6. NFS and ACLs
16. Troubleshooting GlusterFS
16.1. Managing GlusterFS Logs
16.1.1. Setting the Log Directory
16.1.2. Rotating Logs
16.2. Troubleshooting Geo-replication
16.2.1. Locating Log Files
16.2.2. Synchronization is not complete
16.2.3. Issues in Data Synchronization
16.2.4. Geo-replication status displays Faulty very often
16.2.5. Intermediate Master goes to Faulty State
16.3. Troubleshooting POSIX ACLs
16.3.1. setfacl command fails with “setfacl: <file or directory name>: Operation not supported” error
16.4. Troubleshooting NFS
16.4.1. mount command on NFS client fails with “RPC Error: Program not registered”
16.4.2. NFS server start-up fails with “Port is already in use” error in the log file."
16.4.3. mount command fails with “rpc.statd” related error message
16.4.4. mount command takes too long finish.
16.4.5. NFS server glusterfsd starts but initialization fails with “nfsrpc- service: portmap registration of program failed” error message in the log.
16.4.6. mount command fails with NFS server failed error.
16.4.7. showmount fails with clnt_create: RPC: Unable to receive
16.4.8. Application fails with "Invalid argument" or "Value too large for defined data type" error.
17. Command Reference
17.1. gluster Command
17.2. glusterd Daemon
18. Glossary
A. Revision History

Preface

This guide describes how to install and configure Red Hat Storage Software Appliance. It also describes the minimum requirements, and provides step-by-step instructions to install the software in your environment.

1. Audience

The Red Hat Storage Software Appliance User Guide is intended for anyone responsible for installing the software on bare-metal (that is, a non-virtualized physical machine) hardware. This document assumes that you are familiar with the Linux operating system, concepts of File System, and GlusterFS concepts.

2. License

The Red Hat Storage Software Appliance End User License Agreement (EULA) is available at www.redhat.com/licenses/rhel_rha_eula.html

3. Document Conventions

This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set. The Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later includes the Liberation Fonts set by default.

3.1. Typographic Conventions

Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keycaps and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a keycap, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from keycaps by the hyphen connecting each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to the first virtual terminal. Press Ctrl+Alt+F1 to return to your X-Windows session.
The first paragraph highlights the particular keycap to press. The second highlights two key combinations (each a set of three keycaps with each set pressed simultaneously).
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog box text; labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example:
Choose System → Preferences → Mouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, click the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose Applications → Accessories → Character Map from the main menu bar. Next, choose Search → Find… from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose Edit → Paste from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh username@domain.name at a shell prompt. If the remote machine is example.com and your username on that machine is john, type ssh john@example.com.
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release.
Note the words in bold italics above — username, domain.name, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.

3.2. Pull-quote Conventions

Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus: 
books        Desktop   documentation  drafts  mss    photos   stuff  svn
books_tests  Desktop1  downloads      images  notes  scripts  svgs
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows: 
package org.jboss.book.jca.ex1;

import javax.naming.InitialContext;

public class ExClient
{
   public static void main(String args[]) 
       throws Exception
   {
      InitialContext iniCtx = new InitialContext();
      Object         ref    = iniCtx.lookup("EchoBean");
      EchoHome       home   = (EchoHome) ref;
      Echo           echo   = home.create();

      System.out.println("Created Echo");

      System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
   }
}

3.3. Notes and Warnings

Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.

Note

Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.

Important

Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' will not cause data loss but may cause irritation and frustration.

Warning

Warnings should not be ignored. Ignoring warnings will most likely cause data loss.

4. Getting Help and Giving Feedback

4.1. Do You Need Help?

If you experience difficulty with a procedure described in this documentation, visit the Red Hat Customer Portal at http://access.redhat.com. Through the customer portal, you can:
  • search or browse through a knowledgebase of technical support articles about Red Hat products.
  • submit a support case to Red Hat Global Support Services (GSS).
  • access other product documentation.
Red Hat also hosts a large number of electronic mailing lists for discussion of Red Hat software and technology. You can find a list of publicly available mailing lists at https://www.redhat.com/mailman/listinfo. Click on the name of any mailing list to subscribe to that list or to access the list archives.

4.2. We Need Feedback!

If you find a typographical error in this manual, or if you have thought of a way to make this manual better, we would love to hear from you! Please submit a report in Bugzilla: http://bugzilla.redhat.com/ against the product Red Hat Storage Software Appliance.
When submitting a bug report, be sure to mention the manual's identifier: User_Guide
If you have a suggestion for improving the documentation, try to be as specific as possible when describing it. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.

Chapter 1. Introducing Red Hat Storage Software Appliance

The Red Hat Storage Software Appliance 3.2 enables enterprises to treat physical storage as a virtualized, scalable, standardized, scale-on-demand, and centrally managed pool of storage. Enterprises, now have the capability to leverage storage resources the same way they have leveraged computing resources, radically improving storage economics in the process through the use of commodity storage hardware. The appliance's global namespace capability aggregates disk and memory resources into a unified storage volume that is abstracted from the physical hardware. It supports multi-tenancy by partitioning users or groups into logical volumes on shared storage and scales to petabytes of storage capacity.
Red Hat Storage Software Appliance is POSIX-compliant, hence the interface abstracts vendor APIs and the application need not be modified.
Hence, by scaling performance and capacity linearly, you can add capacity as required in matter of few minutes across a wide variety of workloads without affecting the performance. Storage can also be centrally managed across a wide variety of workloads enabling operations to more efficiently manage storage used for a variety of purposes.
The storage software appliance enables users to eliminate their dependence on high cost, difficulty in deployment, and manage monolithic storage arrays. With a storage software appliance, enterprises can now deploy commodity storage hardware and realize superior economics.
Illustration of Red Hat Storage Software Appliance Architecture
Figure 1.1. Red Hat Storage Software Appliance Architecture

The heart of Red Hat Storage Software Appliance is GlusterFS; an open source distributed filesystem distinguished by multiple architectural differences, including a modular, stackable design and a unique, no-metadata server architecture. Eliminating the metadata server provides better performance, improved linear scalability, and increased reliability.

Chapter 2. Preparing to Install Red Hat Storage Software Appliance

2.1. Red Hat Storage Software Appliance Installation Overview
2.2. Checking Red Hat Storage Software Appliance Minimum Requirements
This chapter provides an overview of how you can install Red Hat Storage Software Appliance 3.2, including its minimum hardware, platform requirements, and prerequisites.

2.1. Red Hat Storage Software Appliance Installation Overview

This section provides an overview of how you can install Red Hat Storage Software Appliance.
To install Red Hat Storage Software Appliance, perform the following steps:
  1. Verify that your system matches the minimum requirements. For more information, refer Section 2.2, “Checking Red Hat Storage Software Appliance Minimum Requirements”.
  2. Complete the installation procedure. For more information on installation methods, refer Chapter 4, Installing and Updating Red Hat Storage Software Appliance.

Note

If you wish to install using USB stick, refer to Chapter 3, Downloading and Preparing Installation Media.

2.2. Checking Red Hat Storage Software Appliance Minimum Requirements

Before you install Red Hat Storage Software Appliance, you must verify that your environment matches the minimum requirements described in this section.
General Configuration Considerations
The system must meet the following general requirements:
  • Centralized time servers are available (required in clustered environments)
    For example, ntpd - Network Time Protocol (NTP) Daemon
File System Requirements
Red Hat recommends XFS when formatting the disk sub-system. XFS supports metadata journaling, which facilitates quicker crash recovery. The XFS file system can also be defragmented and enlarged while mounted and active.
For existing Gluster Storage Software Appliance customers who are upgrading to Red Hat Storage Software Appliance, Ext3 and Ext4 file systems is supported.
Cluster Requirements
  • Minimum of four SSA nodes and maximum of 64 SSA nodes.
    Larger configurations are supported, but requires an exception.
  • Initial cluster deployment can be of heterogeneous nodes.
  • Configuration upgrades can support a mixture of node sizes.
    For example, adding node with 4TB drives to a cluster with 2TB drives. However, in replicated configuration, nodes must be added in pairs such that a node and it's replica are the same size.
    Depending on whether the cluster is used for High Performace Computing (HPC), General Purpose, or Archival, the table below lists the supported configurations.
Component HPC General Purpose Archival
Chassis (only applicable with SuperMicro) 2u 24x2.5" Hotswap with redundant power 2u 12x3.5" Hotswap with redundant power 4u 36x3.5" Hotswap with redundant power
Processor Dual Socket Hexacore Xeon Dual Socket Hexacore Xeon Dual Socket Hexacore Core Xeon
Disk 24x 2.5" 15K RPM SAS 12x 3.5" or 24x 2.5" SFF 6gb/s SAS 36x 3.5" 3gb/s SATA II
minimum RAM 48 GB 32 GB 16 GB
Networking 2x10 GigE 2x10 GigE (preferred) or 2x1GigE 2x10 GigE (preferred) or 2x1 GigE
Max # of JBOD attachments 0 2 4
Supported Dell Model R510 R510 R510
Supported HP Model DL-180, DL-370, DL-380 DL-180, DL-370, DL-380 DL-180
JBOD Support NA Dell MD-1200, HP D-2600, HP D-2700 Dell MD-1200, HP D-2600, HP D-2700

Note

The boot device must be 1.4 GB or higher.
All data disks are configured in groups of 12 drives in RAID6 configuration. Infiniband support on exception basis only.
Networking Requirements
Verify either of the following:
  • Gigabit Ethernet
  • 10 Gigabit Ethernet
Compatible Hardware
For successful installation of Red Hat Storage Software Appliance 3.2, you must select your hardware from the Supported Dell, Supported HP, or Supported SuperMicro list of models.
Dell Supported Configurations
Table 2.1. Dell Supported Configurations
Component Recommended Supported Unsupported
Chassis Redundant power configuration R510, R710 (Intel® 5520 Chipset) All other Dell models by exception only
Processor
Dual Six- core processors:
  • Intel® Xeon® X5690 - 3.46GHz
  • Intel® Xeon® X5680 - 3.33GHz
  • Intel® Xeon® X5675 - 3.06GHz
  • Intel® Xeon® X5660 - 2.80GHz
  • Intel® Xeon® X5650 - 2.66GHz
  • Intel® Xeon® E5649 - 2.53GHz
  • Intel® Xeon® E5645 - 2.40GHz
  • Intel® Xeon® L5640 - 2.26GHz
    (also any faster versions of six-core processors)
Unsupported processors:
  • Quad-core processors
  • Single socket configurations
  • AMD based servers
Memory 32GB 24GB Min, 64GB Max
NIC
RAID PERC 6/E SAS 1gb/512, PERC H800 1gb/512 Dell single channel ultra SCSI
System Disk 2x200GB Min (mirrored) 7.2K or 10/15
Data Disk
  • SSD
  • SFF Drives

HP Supported Configurations
Table 2.2. HP Supported Configurations
Component Recommended Supported Unsupported
Chassis
  • Either Model
  • Redundant power configuration
DL-180 G6, DL-370 G7, DL-380 G7 (Intel® 5520 Chipset) All other HP models by exception only
Processor Dual Six-core processors
  • Intel® Xeon® X5690 - 3.46GHz
  • Intel® Xeon® X5680 - 3.33GHz
  • Intel® Xeon® X5675 - 3.06GHz
  • Intel® Xeon® X5660 - 2.80GHz
  • Intel® Xeon® X5650 - 2.66GHz
  • Intel® Xeon® E5649 - 2.53GHz
  • Intel® Xeon® E5645 - 2.40GHz
  • Intel® Xeon® L5640 - 2.26GHz
    (also any faster versions of six-core processors)
  • Quad-core processors
  • Single socket configurations
  • AMD based servers
Memory 32GB 16GB Min, 128GB Max
NIC
RAID
  • HP Smart Array P410/512 with FBWC
  • Smart Array Advanced Pack (SAAP)
  • HP Smart Array P410/256 with FBWC or
  • HP Smart Array P410/512 with FBWC
  • Smart Array Advanced Pack (SAAP)
  • HP Smart Array B110i
  • HP Smart Array P212
  • HP Smart Array P410 with BBWC
System Disk
  • 2x
  • 200GB Min (mirrored) 7.2K or 10/15
Data Disk
  • SSD
  • SFF Drives

Chapter 3. Downloading and Preparing Installation Media

This chapter describes the steps on how you can prepare a bootable USB stick of Red Hat Storage Software Appliance.
You should follow these steps only if your environment needs a bootable USB. Otherwise, you can directly install Red Hat Storage Software Appliance using either an .iso image or booting from PXE. For more information, refer Section 4.2, “Installing from .ISO Image” or Section 4.3, “Booting From PXE”.

Note

You must ensure that the filesystem of the USB stick should be FAT32.
To download and prepare your installation media, perform the following steps.
  1. Download Red Hat Storage Software Appliance from Red Hat Customer Portal
  2. Launch UNetbootin tool.
    UNetbootin Tool
    Figure 3.1. UNetbootin Tool

    Click Diskimage, choose ISO as installation type and browse to select the downloaded .iso image. In the Type field, choose USB Drive.
  3. Click OK. The progress of the installation is displayed.
    Installation Progress Window
    Figure 3.2. Installation Progress Window

    After successfully completing the preparation of USB stick, the following screen will be displayed.
    Installation Completion Window
    Figure 3.3. Installation Completion Window

  4. Click Reboot Now to reboot your installation media.
  5. Copy the .iso image to the USB stick using the following command:
    # cp iso_filename /media/USB/
    You can use your USB stick for installing Red Hat Storage Software Appliance on a server, by selecting the USB boot option in the BIOS boot window.

Chapter 4. Installing and Updating Red Hat Storage Software Appliance

4.1. Installing From USB Stick
4.2. Installing from .ISO Image
4.3. Booting From PXE
4.4. Registering to Red Hat Network (RHN)
4.5. Setting up Software Updates
This chapter describes three different methods you can use for installing Red Hat Storage Software Appliance. You can install Red Hat Storage Software Appliance using an USB stick, an iso image, or boot over PXE.

4.1. Installing From USB Stick

To install Red Hat Storage Software Appliance from the USB stick, perform the following steps:
  1. Insert the USB stick and boot your system. The Software Storage Appliance installation process launches automatically.
    The boot screen
    Figure 4.1. The UNetbootin screen

  2. After an automatic boot delay, the Installation Method screen appears. On this screen, you can choose the type of installation. For USB based installation, select Hard drive and choose appropriate USB stick.
    Installation Method Screen
    Figure 4.2. Installation Method screen

  3. Set the Root Password window is displayed. Setting up a root account and password is one of the most important steps during your installation.
    The root account is used to install packages, upgrade RPMs, and perform most system maintenance. Logging in as root gives you complete control over your system.
    Set Root Password Window
    Figure 4.3. Set Root Password window

    The installation program prompts you to set a root password for your system. You cannot proceed to the next stage of the installation process without entering a root password.
    Click Next to continue.
  4. The Partitioning Type window appears. On this window, you can choose the type of partition. The first four options allow you to perform an automated installation without having to partition your storage devices yourself. If you do not feel comfortable with partitioning your system, choose one of these options and let the installation program partition the storage devices for you. You can choose Create Custom Layout option to partition storage devices manually and create customized layouts.
    Partitioning Type Window
    Figure 4.4. Partitioning Type Window

    Click Next to continue.
  5. The Package Installation window appears showing the progress of installation.
    Package Installation Window
    Figure 4.5. Package Installation window

  6. The Complete status window displays status of your installation.
    Complete Status window
    Figure 4.6. Complete Status window

  7. Remove the installation media.
  8. Click Reboot to reboot the server.

4.2. Installing from .ISO Image

To install Red Hat Storage Software Appliance from the .iso image, perform the following steps:
  1. Insert the installation media and boot your system. The Red Hat Software Storage Appliance installation process launches automatically.
    The Boot screen does not prompt you for any input. Press Enter to begin the installation process.
  2. Set the Root Password window is displayed. Setting up a root account and password is one of the most important steps during your installation.
    The root account is used to install packages, upgrade RPMs, and perform most system maintenance. Logging in as root gives you complete control over your system.
    The installation program prompts you to set a root password for your system. You cannot proceed to the next stage of the installation process without entering a root password.
    Click Next to continue.
  3. The Partitioning Type screen appears. On this screen, you can choose the type of partition. The first four options allow you to perform an automated installation without having to partition your storage devices yourself. If you do not feel comfortable with partitioning your system, choose one of these options and let the installation program partition the storage devices for you. You can choose Create Custom Layout option to partition storage devices manually and create customized layouts.
    Click Next to continue.
  4. The Package Installation screen appears showing the progress of installation.
  5. The Complete screen displays status of your installation.
  6. Click Reboot to reboot the server.

4.3. Booting From PXE

To boot with PXE, you need a properly configured server, and a network interface in your computer that supports PXE.
For more information on how to boot Red Hat Storage Software Appliance from a PXE server, refer to Red Hat Enterprise Linux Installation Guide, Booting from the Network using PXE.

4.4. Registering to Red Hat Network (RHN)

After you have successfully installed Red Hat Storage Software Appliance, you must register with RHN to receive software updates.
To register with RHN, perform the following setps:
  1. In your terminal, run the following command:
    # rhn_register
  2. After you enter your login credentials, the following screen is displayed.
    Registering Screen
    Figure 4.7. Registering Screen

    Important

    In the above screen, you must ensure to choose Limited Updates Only, while registering to RHN. This will ensure that you receive periodic updates of Red Hat Storage Software Appliance.

4.5. Setting up Software Updates

Red Hat strongly recommends you to update your Red Hat Storage Software Appliance on a regular basis with the latest security patches and upgrades. This will ensure that your computer is up-to-date with security updates and upgrades.
To install updates periodically, use the following command
# yum update

Chapter 5. Migrating to Red Hat Storage Software Appliance

5.1. Migration Overview
5.2. Pre-Migration Task
5.3. Migrating your Storage Software Appliance
5.4. Validating your Migration
This chapter documents the migration process of Gluster Storage Software Appliance, Red Hat Enterprise Linux 5.x and CentOS 5.x versions to Red Hat Storage Software Appliance by highlighting key behavioral changes worthy of note when migrating.

5.1. Migration Overview

This section describes the migration overview. The process of migration can be done in three steps:
  • Back up your existing configuration data.
  • Install the Red Hat Storage Software Appliance.
  • Restore the configuration data on your newly installed Red Hat Storage Software Appliance to use your existing data in the volumes.

5.2. Pre-Migration Task

Pre-Migration task is a very important activity that you should not skip to ensure a successful migration.
You must stop all the gluster processes and glusterd daemon, using the following commands:
To stop volume, enter the following command:
# gluster volume stop VOLNAME
To stop glusterd, enter the following command:
# /etc/init.d/glusterd stop

5.3. Migrating your Storage Software Appliance

Important

You must ensure to execute the following migration procedure on all the nodes in your cluster environment.
To migrate your old Storage Software Appliance to the new Red Hat Storage Software Appliance, perform the following steps:
  1. Stop the gluster service using the following command:
    # service glusterd stop
  2. Back up your existing configuration data.
    You must ensure to backup everything in /etc/ directory, especially /etc/glusterd.
  3. Install Red Hat Storage Software Appliance on the system with same network credentials. with the same IP Address and Hostname. For information, see Chapter 4, Installing and Updating Red Hat Storage Software Appliance chapter.

    Warning

    If the IP Address and Hostname are not the same as before, you will not be able to access all the data present in your earlier environment.
    During installation, while creating layout, you must choose create custom layout and proceed with installation. If you choose replace existing linux system option, it formats the entire machine including the HDDs attached to it and erases existing data.
  4. After installation, the system automatically starts glusterd. Hence stop the gluster service using the following command:
    # service glusterd stop
  5. Register to Red Hat Network using the following command:
    # rhn_register
    For more information, see Section 4.4, “Registering to Red Hat Network (RHN)”.
  6. Move your default gluster configuration from /etc/gluserd to a temporary location using the following command:
    # mv /etc/glusterd to /tmp/etc/glusterd.old
  7. Copy the /etc/glusterd directory from the backup (refer Step 2) to your new installation.
  8. Re-mount data disks at the same mount locations as before.
    # mount /data/disk
  9. Add entries to /etc/fstab to mount data disks at the same mount point as before.

    Note

    You must ensure that mount points exists in your cluster environment.
  10. Mount all data disks using the following command:
    # mount -a
  11. If you have taken a backup of any other directories from /etc/, you must update the system configuration files into your new installation.
  12. You have now successfully migrated your environment to Red Hat Storage Software Appliance.
  13. You must restart glusterd management daemon using the following command:
    # service glusterd start
  14. Restart glusterfsd process using the following command:
    # gluster volume start volname force

5.4. Validating your Migration

After completing your migration process, you must reboot your system to ensure all services have started automatically and your volumes are mounted.
For migration procedures other than the above documented scenario, contact the Red Hat Support.

Chapter 6. Starting and Stopping the glusterd Daemon

6.1. Starting and Stopping glusterd Manually
6.2. Starting glusterd Automatically
The glusterd daemon serves as the Gluster elastic volume manager, overseeing glusterfs processes, and co-ordinating dynamic volume operations, such as adding and removing volumes across multiple storage servers non-disruptively.
This section describes how to start the glusterd daemon in the following ways:

Note

You must start glusterd on all servers.

6.1. Starting and Stopping glusterd Manually

This section describes how to start and stop glusterd manually
  • To start glusterd manually, enter the following command:
    # /etc/init.d/glusterd start
  • To stop glusterd manually, enter the following command:
    # /etc/init.d/glusterd stop

6.2. Starting glusterd Automatically

To automatically start the glusterd daemon every time the system boots, enter the following from the command line:
# chkconfig glusterd on

Chapter 7. Gluster Console Manager – Command Line Utility for Configuration and Management

GlusterFS offers a single command line utility known as the Gluster Console Manager to simplify configuration and management of your storage environment. The Gluster Console Manager provides functionality similar to the LVM (Logical Volume Manager) CLI or ZFS Command Line Interface, but across multiple storage servers. You can use the Gluster Console Manager online, while volumes are mounted and active.
Using the Gluster Console Manager, administrators can create new volumes, start volumes, and stop volumes, as required. Administrators can also add bricks to volumes, remove bricks from existing volumes, as well as change translator settings, among other operations.
Gluster automatically synchronizes volume configuration information across all Gluster servers. Administrators can also use the commands to create scripts for automation, as well as use the commands as an API to allow integration with third-party applications.
Running the Gluster Console Manager
You can run the Gluster Console Manager on any Gluster storage server. You can run Gluster commands either by invoking the commands directly from the shell, or by running the Gluster CLI in interactive mode.
  • To run commands directly from the shell:
    # gluster peer command
    For example:
    # gluster peer status
  • To run the Gluster Console Manager in interactive mode
    # gluster
    Upon invoking the Console Manager, you will get an interactive shell where you can execute gluster commands.
    gluster> command
    For example:
    # gluster
    gluster > peer status

Note

You can use the gluster command over SSH for remote execution.

Chapter 8. Trusted Storage Pools – Preparing GlusterFS for Management

8.1. Adding Servers to Trusted Storage Pool
8.2. Removing Server from the Trusted Storage Pool
Before configuring a GlusterFS volume, you need to create a trusted storage pool consisting of the storage servers that will make up the volume.
A storage pool is a trusted network of storage servers. When you start the first server, the storage pool consists of that server alone. To add additional storage servers to the storage pool, you can use the probe command from a storage server that is already trusted.

Note

The GlusterFS service must be running on all storage servers that you want to add to the storage pool. For more information, see Chapter 6, Starting and Stopping the glusterd Daemon

8.1. Adding Servers to Trusted Storage Pool

To add servers to the trusted storage pool
  1. Ensure that the host names used to create the storage pool are resolvable by DNS before probing the servers.
    To add a server to the storage pool, use the following command:
    # gluster peer probe server
    For example, to create a trusted storage pool of four servers, add three servers to the storage pool from server1: 
    # gluster peer probe server2
Probe successful

# gluster peer probe server3
Probe successful

# gluster peer probe server4
Probe successful

    Note

    Do not self-probe the first server/localhost (server1).
  2. Verify the peer status from the first server using the following commands: 
    # gluster peer status
Number of Peers: 3

Hostname: server2
Uuid: 5e987bda-16dd-43c2-835b-08b7d55e94e5
State: Peer in Cluster (Connected)

Hostname: server3
Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7
State: Peer in Cluster (Connected)

Hostname: server4
Uuid: 3e0caba-9df7-4f66-8e5d-cbc348f29ff7
State: Peer in Cluster (Connected)

8.2. Removing Server from the Trusted Storage Pool

To remove a server to the storage pool
To remove a server to the storage pool, use the following command:
# gluster peer detach server
For example, to remove server4 from the trusted storage pool, run the following command: 
# gluster peer detach server4
Detach successful

Chapter 9. Setting Up GlusterFS Server Volumes

9.1. Creating Distributed Volumes
9.2. Creating Replicated Volumes
9.3. Creating Striped Volumes
9.4. Creating Distributed Striped Volumes
9.5. Creating Distributed Replicated Volumes
9.6. Starting Volumes
9.7. Displaying Volume Information
A volume is a logical collection of bricks where each brick is an export directory on a server in the trusted storage pool. Most of the gluster management operations happen on the volume.
You can create new volumes in your storage environment, as needed. When creating a new volume, you must specify the bricks that comprise the volume.
To create a new volume

9.1. Creating Distributed Volumes

Distributed volumes distribute files throughout the bricks in the volume. You can use distributed volumes where the requirement is to scale storage and the redundancy is either not important or is provided by other hardware/software layers.

Note

Disk/server failure in distributed volumes can result in a serious loss of data since directory contents are spread randomly throughout the bricks in the volume.
To create a distributed volume
  1. Create a trusted storage pool consisting of the storage servers that will comprise the volume.
    For more information, seeSection 8.1, “Adding Servers to Trusted Storage Pool”.
  2. Create the volume using the following command:
    # gluster volume create NEW-VOLNAME [transport tcp | rdma | tcp,rdma] NEW-BRICK...
    For example, to create a distributed volume with four storage servers using tcp: 
    # gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
Creation of test-volume has been successful
Please start the volume to access data.
    You can optionally display the volume information using the following command: 
    # gluster volume info
Volume Name: test-volume
Type: Distribute
Status: Created
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4
    To create a distributed volume with four storage servers over InfiniBand: 
    # gluster volume create test-volume transport rdma server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
Creation of test-volume has been successful
Please start the volume to access data.

    Note

    If transport type is not specified, it defaults to tcp.
  3. (Optional) Set additional options if required, such as auth.allow or auth.reject.
    For example:
    # gluster volume set test-volume auth.allow 10.*
    For more information, see Section 11.1, “Tuning Volume Options”.

    Note

    Make sure you start your volumes before you try to mount them or else client operations after the mount will hang, see Section 9.6, “Starting Volumes ” for details.
  4. (Optional) Display the volume information to confirm that the volume has started using the following command: 
    # gluster volume info
Volume Name: test-volume
Type: Distribute
Status: Created
Number of Bricks: 4
Transport-type: rdma
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4

9.2. Creating Replicated Volumes

Replicated volumes replicate files throughout the bricks in the volume. You can use replicated volumes in environments where high-availability and high-reliability are critical.
To configure a replicated volume
  1. Create a trusted storage pool consisting of the storage servers that will comprise the volume.
    For more information, see Section 8.1, “Adding Servers to Trusted Storage Pool” .
  2. Create the volume using the following command:

    Note

    The number of bricks should be equal to of the replica count for a replicated volume. To protect against server and disk failures, it is recommended that the bricks of the volume are from different servers.
    # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma tcp,rdma] NEW-BRICK...
    For example, to create a replicated volume with two storage servers: 
    # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2
Creation of test-volume has been successful
Please start the volume to access data.
  3. (Optional) Set additional options if required, such as auth.allow or auth.reject.
    For example:
    # gluster volume set test-volume auth.allow 10.*

    Note

    Make sure you start your volumes before you try to mount them or else client operations after the mount will hang, see Section 9.6, “Starting Volumes ” for details.

9.3. Creating Striped Volumes

Stripes data across bricks in the volume. For best results, you should use striped volumes only in high concurrency environments accessing very large files.
To configure a striped volume
  1. Create a trusted storage pool consisting of the storage servers that will comprise the volume.
    For more information, see Section 8.1, “Adding Servers to Trusted Storage Pool”.
  2. Create the volume using the following command:

    Note

    The number of bricks should be a equal to the stripe count for a striped volume.
    # gluster volume create NEW-VOLNAME [stripe COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
    For example, to create a striped volume across two storage servers: 
    # gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2
Creation of test-volume has been successful
Please start the volume to access data.
  3. (Optional) Set additional options if required, such as auth.allow or auth.reject.
    For example:
    # gluster volume set test-volume auth.allow 10.*

    Note

    Make sure you start your volumes before you try to mount them or else client operations after the mount will hang, see Section 9.6, “Starting Volumes ” for details.

9.4. Creating Distributed Striped Volumes

Distributed striped volumes stripe data across two or more nodes in the cluster. For best results, you should use distributed striped volumes where the requirement is to scale storage and in high concurrency environments accessing very large files is critical.
To configure a distributed striped volume
  1. Create a trusted storage pool consisting of the storage servers that will comprise the volume.
    For more information, see Section 8.1, “Adding Servers to Trusted Storage Pool”.
  2. Create the volume using the following command:

    Note

    The number of bricks should be a multiple of the stripe count for a distributed striped volume.
    # gluster volume create NEW-VOLNAME [stripe COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
    For example, to create a distributed striped volume across four storage servers: 
    # gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
Creation of test-volume has been successful
Please start the volume to access data.
    For example, to create a distributed striped volume across eight storage servers: 
    # gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8
Creation of test-volume has been successful
Please start the volume to access data.
  3. (Optional) Set additional options if required, such as auth.allow or auth.reject.
    For example:
    # gluster volume set test-volume auth.allow 10.*

    Note

    Make sure you start your volumes before you try to mount them or else client operations after the mount will hang, see Section 9.6, “Starting Volumes ” for details.

9.5. Creating Distributed Replicated Volumes

Distributes files across replicated bricks in the volume. You can use distributed replicated volumes in environments where the requirement is to scale storage and high-reliability is critical. Distributed replicated volumes also offer improved read performance in most environments.
To configure a distributed replicated volume
  1. Create a trusted storage pool consisting of the storage servers that will comprise the volume.
    For more information, see Section 8.1, “Adding Servers to Trusted Storage Pool”.
  2. Create the volume using the following command:

    Note

    The number of bricks should be a multiple of the replica count for a distributed replicated volume. Also, the order in which bricks are specified has a great effect on data protection. Each replica_count consecutive bricks in the list you give will form a replica set, with all replica sets combined into a volume-wide distribute set. To make sure that replica-set members are not placed on the same node, list the first brick on every server, then the second brick on every server in the same order, and so on.
    # gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK...
    For example, four node distributed (replicated) volume with a two-way mirror: 
    # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4
Creation of test-volume has been successful
Please start the volume to access data.
    For example, to create a six node distributed (replicated) volume with a two-way mirror: 
    # gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6
Creation of test-volume has been successful
Please start the volume to access data.
  3. (Optional) Set additional options if required, such as auth.allow or auth.reject.
    For example:
    # gluster volume set test-volume auth.allow 10.*

    Note

    Make sure you start your volumes before you try to mount them or else client operations after the mount will hang, see Section 9.6, “Starting Volumes ” for details.

9.6. Starting Volumes

You must start your volumes before you try to mount.
To start a volume
  • Start the volume using the following command:
    # gluster volume start VOLNAME
    For example, to start test-volume: 
    # gluster volume start test-volume
Starting test-volume has been successful

9.7. Displaying Volume Information

You can display information about a specific volume, or all volumes, as needed.
To display volume information
  • Display information about a specific volume using the following command:
    # gluster volume info VOLNAME
    For example, to display information about test-volume: 
    # gluster volume info test-volume
Volume Name: test-volume
Type: Distribute
Status: Created
Number of Bricks: 4
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4
  • Display information about all volumes using the following command:
    # gluster volume info all 
    # gluster volume info all

Volume Name: test-volume
Type: Distribute
Status: Created
Number of Bricks: 4
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4

Volume Name: mirror
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 X 2 = 4
Bricks:
Brick1: server1:/brick1
Brick2: server2:/brick2
Brick3: server3:/brick3
Brick4: server4:/brick4

Volume Name: Vol
Type: Distribute
Status: Started
Number of Bricks: 1
Bricks:
Brick: server:/brick6

Chapter 10. Accessing Data - Setting Up GlusterFS Client

10.1. Gluster Native Client
10.1.1. Installing the Gluster Native Client
10.1.2. Mounting Volumes
10.2. NFS
10.2.1. Using NFS to Mount Volumes
10.3. CIFS
10.3.1. Using CIFS to Mount Volumes
Gluster offers multiple ways for users to access gluster volumes:
  • Gluster Native Client - This method provides high concurrency, performance and transparent failover in GNU/Linux clients. The Gluster Native Client is POSIX conformant. For accessing volumes using gluster native protocol, you need to install gluster native client. For more information, see Section 10.1, “Gluster Native Client”.
  • NFS – This method provides access to gluster volumes with NFS v3. Extensive testing has be done on GNU/Linux clients and NFS implementation in other operating system, such as FreeBSD, and Mac OS X, as well as Windows 7 (Professional and Up), Windows Server 2003, and others, may work with gluster NFS server implementation.
    For more information, see Section 10.2, “NFS”.
  • CIFS - This method provides access to volumes when using Microsoft Windows as well as SAMBA clients. For this access method, Samba packages need to be present on the client side. For more information, see Section 10.3, “CIFS”.

10.1. Gluster Native Client

The Gluster Native Client is a POSIX conformant, FUSE-based, client running in user space. Gluster Native Client is the recommended method for accessing volumes when high concurrency and high write performance is required.
This section introduces the Gluster Native Client and explains how to install the software on client machines. This section also describes how to mount volumes on clients (both manually and automatically) and how to verify that the volume has mounted successfully.

10.1.1. Installing the Gluster Native Client

This section describes how to install Gluster Native Client and includes the following topics:
  • Before Installation
  • Installing on Red Hat Package Manager (RPM) Distributions
  • Performing a Source Installation

10.1.1.1. Before Installation

Before you begin installing the Gluster Native Client, you need to verify that the FUSE module is loaded on the client.
To verify that the FUSE module is installed
  1. Add the FUSE loadable kernel module (LKM) to the Linux kernel using the following command:
    # modprobe fuse
  2. Verify that the FUSE module is loaded using the following command:
    # dmesg | grep -i fuse
    fuse init (API version 7.13)

10.1.1.2. Installing on Red Hat Package Manager (RPM) Distributions

To install Gluster Native Client on RPM distribution-based systems
  1. Install required prerequisites on the client using the following command:
    $ sudo yum -y install openssh-server wget fuse fuse-libs openib libibverbs
  2. Ensure that TCP and UDP ports 24007 and 24008 are open on all Gluster servers. Apart from these ports, you need to open one port for each brick starting from port 24009. For example: if you have five bricks, you need to have ports 24009 to 24014 open.
    You can use the following chains with iptables:
    $ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -j ACCEPT
    $ sudo iptables -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24009:24014 -j ACCEPT

    Note

    If you already have iptable chains, make sure that the above ACCEPT rules precede the DROP rules. This can be achieved by providing a lower rule number than the DROP rule.
  3. Download the latest GlusterFS core, FUSE, and RDMA RPM files to each client.
    The core package contains the Gluster Native Client. The optional FUSE package contain the FUSE translator for glusterfs native mounting on client systems and the RDMA packages contain OpenFabrics verbs RDMA module for Infiniband.
    You can download the software at http://download.gluster.com/pub/gluster/glusterfs/3.2/LATEST/.
  4. For each RPM file, get the checksum (using the following command) and compare it against the checksum for that file in the md5sum.
    $ md5sum RPM_file.rpm
    The md5sum of the packages is available at http://download.gluster.com/pub/gluster/glusterfs/3.2/LATEST/.
  5. Install Gluster Native Client on the client.
    Use the following commands:
    $ sudo rpm -Uvh core_RPM_file
    $ sudo rpm -Uvh fuse_RPM_file
    $ sudo rpm -Uvh rdma_RPM_file
    For example:
    $ sudo rpm -Uvh glusterfs-core-3.2.X.x86_64.rpm
    $ sudo rpm -Uvh glusterfs-fuse-3.2.X.x86_64.rpm
    $ sudo rpm -Uvh glusterfs-rdma-3.2.X.x86_64.rpm

    Note

    The RDMA module is only required when using Infiniband.

10.1.1.3. Performing a Source Installation

To build and install Gluster Native Client from the source code
  1. Create a new directory using the following commands:
    # mkdir glusterfs
    # cd glusterfs
  2. Download the source code.
    You can download the source at http://download.gluster.com/pub/gluster/glusterfs/3.2/LATEST/.
  3. Extract the source code using the following command:
    # tar -xvzf SOURCE-FILE
  4. Run the configuration utility using the following command:
    # ./configure
    GlusterFS configure summary
    ==================
    FUSE client : yes
    Infiniband verbs : yes
    epoll IO multiplex : yes
    argp-standalone : no
    fusermount : no
    readline : yes
    The configuration summary shows the components that will be built with Gluster Native Client.
  5. Build the Gluster Native Client software using the following commands:
    # make
    # make install
  6. Verify that the correct version of Gluster Native Client is installed, using the following command:
    # glusterfs –-version

10.1.2. Mounting Volumes

After installing the Gluster Native Client, you need to mount Gluster volumes to access data. There are two methods you can choose:
After mounting a volume, you can test the mounted volume using the procedure described in Section 10.1.2.3, “Testing Mounted Volumes”.

Note

Server names selected during creation of Volumes should be resolvable in the client machine. You can use appropriate /etc/hosts entries or DNS server to resolve server names to IP addresses.

10.1.2.1. Manually Mounting Volumes

To manually mount a Gluster volume
  • To mount a volume, use the following command:
    # mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR
    For example:
    # mount -t glusterfs server1:/test-volume /mnt/glusterfs

    Note

    The server specified in the mount command is only used to fetch the gluster configuration volfile describing the volume name. Subsequently, the client will communicate directly with the servers mentioned in the volfile (which might not even include the one used for mount).
    If you see a usage message like "Usage: mount.glusterfs", mount usually requires you to create a directory to be used as the mount point. Run "mkdir /mnt/glusterfs" before you attempt to run the mount command listed above.
Mounting Options
You can specify the following options when using the mount -t glusterfs command. Note that you need to separate all options with commas.
log-level=loglevel
log-file=logfile
transport=transport-type
direct-io-mode=[enable|disable]
For example:
# mount -t glusterfs -o log-level=WARNING,log-file=/var/log/gluster.log server1:/test-volume /mnt/glusterfs

10.1.2.2. Automatically Mounting Volumes

You can configure your system to automatically mount the Gluster volume each time your system starts.
To automatically mount a Gluster volume
  • To mount a volume, edit the /etc/fstab file and add the following line:
    HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR glusterfs defaults,_netdev 0 0
    For example:
    server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0
Mounting Options
You can specify the following options when updating the /etc/fstab file. Note that you need to separate all options with commas.
log-level=loglevel
log-file=logfile
transport=transport-type
direct-io-mode=[enable|disable]
For example:
HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR glusterfs defaults,_netdev,log-level=WARNING,log-file=/var/log/gluster.log 0 0

Note

The server specified in the mount command is only used to fetch the gluster configuration volfile describing the volume name. Subsequently, the client will communicate directly with the servers mentioned in the volfile (which might not even include the one used for mount).

10.1.2.3. Testing Mounted Volumes

To test mounted volumes
  • Use the mount command by entering the following:
    # mount
    For example,
    The output of the mount command on the client will display an entry like the following:
    server1:/test-volume on /mnt/glusterfs type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072
  • Use the df command by entering the following:
    # df
    For example,
    The output of df command on the client will display the aggregated storage space from all the bricks in a volume.
    # df -h /mnt/glusterfs Filesystem Size Used Avail Use% Mounted on server1:/test-volume 28T 22T 5.4T 82% /mnt/glusterfs
  • Change to the directory and list the contents by entering the following:
    # cd MOUNTDIR
    # ls
  • For example,
    # cd /mnt/glusterfs
    # ls

10.2. NFS

This section describes how to use NFS to mount Gluster volumes (both manually and automatically) and how to verify that the volume has been mounted successfully.

Note

NFS does not work with CTDB and NLM due to absence of locking support.

10.2.1. Using NFS to Mount Volumes

You can use either of the following methods to mount Gluster volumes:
After mounting a volume, you can test the mounted volume using the procedure described in Section 10.2.1.3, “Testing Volumes Mounted Using NFS”.

10.2.1.1. Manually Mounting Volumes Using NFS

To manually mount a Gluster volume using NFS
  • To mount a volume, use the following command:
    # mount -t nfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR
    For example:
    # mount -t nfs server1:/test-volume /mnt/glusterfs

    Note

    Gluster NFS server does not support UDP. If the NFS client you are using defaults to connecting using UDP, the following message appears:
    requested NFS version or transport protocol is not supported.
    To connect using TCP
  • Add the following option to the mount command:
    -o mountproto=tcp
    For example:
    # mount -o mountproto=tcp -t nfs server1:/test-volume /mnt/glusterfs
To mount Gluster NFS server from a Solaris client
  • Use the following command:
    # mount -o proto=tcp,vers=3 nfs://HOSTNAME-OR-IPADDRESS:38467/VOLNAME MOUNTDIR
    For example:
    # mount -o proto=tcp,vers=3 nfs://server1:38467/test-volume /mnt/glusterfs

10.2.1.2. Automatically Mounting Volumes Using NFS

You can configure your system to automatically mount Gluster volumes using NFS each time the system starts.
To automatically mount a Gluster volume using NFS
  • To mount a volume, edit the /etc/fstab file and add the following line:
    HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR nfs defaults,_netdev 0 0
    For example,
    server1:/test-volume /mnt/glusterfs nfs defaults,_netdev 0 0

    Note

    Gluster NFS server does not support UDP. If the NFS client you are using defaults to connecting using UDP, the following message appears:
    requested NFS version or transport protocol is not supported.
    To connect using TCP
  • Add the following entry in /etc/fstab file :
    HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR nfs defaults,_netdev,mountproto=tcp 0 0
    For example,
    server1:/test-volume /mnt/glusterfs nfs defaults,_netdev,mountproto=tcp 0 0
To automount NFS mounts
Gluster supports *nix standard method of automounting NFS mounts. Update the /etc/auto.master and /etc/auto.misc and restart the autofs service. After that, whenever a user or process attempts to access the directory it will be mounted in the background.

10.2.1.3. Testing Volumes Mounted Using NFS

You can confirm that Gluster directories are mounting successfully.
To test mounted volumes
  • Use the mount command by entering the following:
    # mount
    For example, the output of the mount command on the client will display an entry like the following:
    server1:/test-volume on /mnt/glusterfs type nfs.glusterfs (rw,allow_other,default_permissions,max_read=131072)
  • Use the df command by entering the following:
    # df
    For example, the output of df command on the client will display the aggregated storage space from all the bricks in a volume. 
    # df -h /mnt/glusterfs
Filesystem Size Used Avail Use% Mounted on
server1:/test-volume 28T 22T 5.4T 82% /mnt/glusterfs
  • Change to the directory and list the contents by entering the following:
    # cd MOUNTDIR
    # ls
    For example,
    # cd /mnt/glusterfs
    # ls

10.3. CIFS

You can export glusterfs mount point as the samba export, and then mount it using CIFS protocol.
This section describes how to mount CIFS shares on Microsoft Windows-based clients (both manually and automatically) and how to verify that the volume has mounted successfully.

Note

CIFS access using the Mac OS X Finder is not supported, however, you can use the Mac OS X command line to access Gluster volumes using CIFS.

10.3.1. Using CIFS to Mount Volumes

You can use either of the following methods to mount Gluster volumes:
After mounting a volume, you can test the mounted volume using the procedure described in Section 10.3.1.4, “Testing Volumes Mounted Using CIFS”.
You can also use Samba for exporting Gluster Volumes through CIFS protocol.

10.3.1.1. Exporting Gluster Volumes Through Samba

We recommend you to use Samba for exporting Gluster volumes through the CIFS protocol.
To export volumes through CIFS protocol
  1. Mount a Gluster volume. For more information on mounting volumes, see Section 10.1.2, “Mounting Volumes”.
  2. Setup Samba configuration to export the mount point of the Gluster volume.
    For example, if a Gluster volume is mounted on /mnt/gluster, you must edit smb.conf file to enable exporting this through CIFS. Open smb.conf file in an editor and add the following lines for a simple configuration:
    [glustertest]
    comment = For testing a Gluster volume exported through CIFS
    path = /mnt/gluster
    read only = no
    guest ok = yes
Save the changes and start the smb service using your systems init scripts. Now the Gluster volume can be mounted using the CIFS protocol.

Note

To be able mount from any server in the trusted storage pool, you must repeat these steps on each Gluster node. For more advanced configurations, see Samba documentation.

10.3.1.2. Manually Mounting Volumes Using CIFS

You can manually mount Gluster volumes using CIFS on Microsoft Windows-based client machines.
To manually mount a Gluster volume using CIFS
  1. Using Windows Explorer, choose Tools > Map Network Drive… from the menu. The Map Network Drive window appears.
  2. Choose the drive letter using the Drive drop-down list.
  3. Click Browse, select the volume to map to the network drive, and click OK.
  4. Click Finish.
The network drive (mapped to the volume) appears in the Computer window.
Alternatively, to manually mount a Gluster volume using CIFS.
  • Click Start > Run and enter the following:
    \\SERVERNAME\VOLNAME
    For example:
    \\server1\test-volume

10.3.1.3. Automatically Mounting Volumes Using CIFS

You can configure your system to automatically mount Gluster volumes using CIFS on Microsoft Windows-based clients each time the system starts.
To automatically mount a Gluster volume using CIFS
The network drive (mapped to the volume) appears in the Computer window and is reconnected each time the system starts.
  1. Using Windows Explorer, choose Tools > Map Network Drive… from the menu. The Map Network Drive window appears.
  2. Choose the drive letter using the Drive drop-down list.
  3. Click Browse, select the volume to map to the network drive, and click OK.
  4. Click the Reconnect at logon checkbox.
  5. Click Finish.

10.3.1.4. Testing Volumes Mounted Using CIFS

You can confirm that Gluster directories are mounting successfully by navigating to the directory using Windows Explorer.

Chapter 11. Managing GlusterFS Volumes

11.1. Tuning Volume Options
11.2. Expanding Volumes
11.3. Shrinking Volumes
11.4. Migrating Volumes
11.5. Rebalancing Volumes
11.5.1. Rebalancing Volume to Fix Layout Changes
11.5.2. Rebalancing Volume to Migrate Existing Data
11.5.3. Rebalancing Volume to Fix Layout and Migrate Existing Data
11.6. Stopping Volumes
11.7. Deleting Volumes
11.8. Triggering Self-Heal on Replicate
This section describes how to perform common GlusterFS management operations, including the following:

11.1. Tuning Volume Options

You can tune volume options, as needed, while the cluster is online and available.

Note

Gluster recommends you to set server.allow-insecure option to ON if there are too many bricks in each volume or if there are too many services which have already utilized all the privileged ports in the system. Turning this option ON allows ports to accept/reject messages from insecure ports. So, use this option only if your deployment requires it.
To tune volume options
  • Tune volume options using the following command:
    # gluster volume set VOLNAME OPTION PARAMETER
    For example, to specify the performance cache size for test-volume: 
    # gluster volume set test-volume performance.cache-size 256MB
Set volume successful
    The following table lists the Volume options along with its description and default value:

    Note

    The default options given here are subject to modification at any given time and may not be the same for all versions.
    Option Description Default Value Available Options
    auth.allow IP addresses/Host name of the clients which should be allowed to access the the volume. * (allow all) Valid IP address which includes wild card patterns including *, such as 192.168.1.*
    auth.reject IP addresses/Host name of the clients which should be denied to access the volume. NONE (reject none)
    cluster.self-heal-window-size Specifies the maximum number of blocks per file on which self-heal would happen simultaneously. 16 0 < data-self-heal-window-size < 1025
    cluster.data-self-heal-algorithm Selects between "full", "diff", and “reset”. The "full" algorithm copies the entire file from source to sinks. The "diff" algorithm copies to sinks only those blocks whose checksums don't match with those of source. Reset uses a heuristic model. If the file does not exist on one of the subvolumes, or a zero-byte file exists (created by entry self-heal) the entire content has to be copied anyway, so there is no benefit from using the "diff" algorithm. If the file size is about the same as page size, the entire file can be read and written with a few operations, which will be faster than "diff" which has to read checksums and then read and write. Reset full/diff
    cluster.min-free-disk Specifies the percentage of disk space that must be kept free. Might be useful for non-uniform bricks. 10% Percentage of required minimum free disk space
    cluster.stripe-block-size Specifies the size of the stripe unit that will be read from or written to. Optionally different stripe unit sizes can be specified for different files, with the following pattern <filename-pattern:blk-size>. 128 KB (for all files) size in bytes
    cluster.self-heal-daemon Allows you to turn-off proactive self-heal on replicated volumes. on On | Off
    diagnostics.brick-log-level Changes the log-level of the bricks. INFO BUG|WARNING|ERROR|CRITICAL|NONE|TRACE
    diagnostics.client-log-level Changes the log-level of the clients. INFO BUG|WARNING|ERROR|CRITICAL|NONE|TRACE
    diagnostics.latency-measurement Statistics related to the latency of each operation would be tracked. off On | Off
    diagnostics.dump-fd-stats Statistics related to file-operations would be tracked. off On | Off
    features.quota-timeout For performance reasons, quota caches the directory sizes on client. You can set timeout indicating the maximum duration of directory sizes in cache, from the time they are populated, during which they are considered valid. 0 0 < 3600 secs
    geo-replication.indexing Use this option to automatically sync the changes in the filesystem from Master to Slave. off On | Off
    network.frame-timeout The time frame after which the operation has to be declared as dead, if the server does not respond for a particular operation. 1800 (30 mins) 1800 secs
    network.ping-timeout The time duration for which the client waits to check if the server is responsive. When a ping timeout happens, there is a network disconnect between the client and server. All resources held by server on behalf of the client get cleaned up. When a reconnection happens, all resources will need to be re-acquired before the client can resume its operations on the server. Additionally, the locks will be acquired and the lock tables updated.
    This reconnect is a very expensive operation and should be avoided.
    		42 Secs 42 Secs
    nfs.enable-ino32 For 32-bit nfs clients or applications that do not support 64-bit inode numbers or large files, use this option from the CLI to make Gluster NFS return 32-bit inode numbers instead of 64-bit inode numbers. Applications that will benefit are those that were either:
    * Built 32-bit and run on 32-bit machines.
    * Built 32-bit on 64-bit systems.
    * Built 64-bit but use a library built 32-bit, especially relevant for python and perl scripts.
    Either of the conditions above can lead to application on Linux NFS clients failing with "Invalid argument" or "Value too large for defined data type" errors.
    		off On | Off
    nfs.volume-access Set the access type for the specified sub-volume. read-write read-write|read-only
    nfs.trusted-write If there is an UNSTABLE write from the client, STABLE flag will be returned to force the client to not send a COMMIT request.
    In some environments, combined with a replicated GlusterFS setup, this option can improve write performance. This flag allows users to trust Gluster replication logic to sync data to the disks and recover when required. COMMIT requests if received will be handled in a default manner by fsyncing. STABLE writes are still handled in a sync manner.
    		off On | Off
    nfs.trusted-sync All writes and COMMIT requests are treated as async. This implies that no write requests are guaranteed to be on server disks when the write reply is received at the NFS client. Trusted sync includes trusted-write behavior. off On | Off
    nfs.export-dir By default, all sub-volumes of NFS are exported as individual exports. Now, this option allows you to export only the specified subdirectory or subdirectories in the volume. This option can also be used in conjunction with nfs3.export-volumes option to restrict exports only to the subdirectories specified through this option. You must provide an absolute path. Enabled for all sub directories. Enable|Disable
    nfs.export-volumes Enable/Disable exporting entire volumes, instead if used in conjunction with nfs3.export-dir, can allow setting up only subdirectories as exports. on On | Off
    nfs.rpc-auth-unix Enable/Disable the AUTH_UNIX authentication type. This option is enabled by default for better interoperability. However, you can disable it if required. on On | Off
    nfs.rpc-auth-null Enable/Disable the AUTH_NULL authentication type. It is not recommended to change the default value for this option. on On | Off
    nfs.rpc-auth-allow<IP- Addresses> Allow a comma separated list of addresses and/or hostnames to connect to the server. By default, all clients are disallowed. This allows you to define a general rule for all exported volumes. Reject All IP address or Host name
    nfs.rpc-auth-reject IP- Addresses Reject a comma separated list of addresses and/or hostnames from connecting to the server. By default, all connections are disallowed. This allows you to define a general rule for all exported volumes. Reject All IP address or Host name
    nfs.ports-insecure Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. off On | Off
    nfs.addr-namelookup Turn-off name lookup for incoming client connections using this option. In some setups, the name server can take too long to reply to DNS queries resulting in timeouts of mount requests. Use this option to turn off name lookups during address authentication. Note, turning this off will prevent you from using hostnames in rpc-auth.addr.* filters. on On | Off
    nfs.register-with- portmap For systems that need to run multiple NFS servers, you need to prevent more than one from registering with portmap service. Use this option to turn off portmap registration for Gluster NFS. on On | Off
    nfs.port <PORT- NUMBER> Use this option on systems that need Gluster NFS to be associated with a non-default port number. 38465- 38467
    nfs.disable Turn-off volume being exported by NFS off On | Off
    performance.write-behind-window-size Size of the per-file write-behind buffer. 1 MB Write-behind cache size
    performance.io-thread-count The number of threads in IO threads translator. 16 0 < io-threads < 65
    performance.flush-behind If this option is set ON, instructs write-behind translator to perform flush in background, by returning success (or any errors, if any of previous writes were failed) to application even before flush is sent to backend filesystem. On On | Off
    performance.cache-max-file-size Sets the maximum file size cached by the io-cache translator. Can use the normal size descriptors of KB, MB, GB,TB or PB (for example, 6GB). Maximum size uint64. 2 ^ 64 -1 bytes size in bytes
    performance.cache-min-file-size Sets the minimum file size cached by the io-cache translator. Values same as "max" above. 0B size in bytes
    performance.cache-refresh-timeout The cached data for a file will be retained till 'cache-refresh-timeout' seconds, after which data re-validation is performed. 1 sec 0 < cache-timeout < 61
    performance.cache-size Size of the read cache. 32 MB size in bytes
    server.allow-insecure Allow client connections from unprivileged ports. By default only privileged ports are allowed. This is a global setting in case insecure ports are to be enabled for all exports using a single option. on On | Off

11.2. Expanding Volumes

You can expand volumes, as needed, while the cluster is online and available. For example, you might want to add a brick to a distributed volume, thereby increasing the distribution and adding to the capacity of the GlusterFS volume.
Similarly, you might want to add a group of bricks to a distributed replicated volume, increasing the capacity of the GlusterFS volume.

Note

When expanding distributed replicated and distributed striped volumes, you need to add a number of bricks that is a multiple of the replica or stripe count. For example, to expand a distributed replicated volume with a replica count of 2, you need to add bricks in multiples of 2 (such as 4, 6, 8, etc.).
To expand a volume
  1. On the first server in the cluster, probe the server to which you want to add the new brick using the following command:
    # gluster peer probe HOSTNAME
    For example: 
    # gluster peer probe server4
Probe successful
  2. Add the brick using the following command:
    # gluster volume add-brick VOLNAME NEW-BRICK
    For example: 
    # gluster volume add-brick test-volume server4:/exp4
Add Brick successful
  3. Check the volume information using the following command:
    # gluster volume info
    The command displays information similar to the following: 
    Volume Name: test-volume
Type: Distribute
Status: Started
Number of Bricks: 4
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server3:/exp3
Brick4: server4:/exp4
  4. Rebalance the volume to ensure that all files are distributed to the new brick.
    You can use the rebalance command as described in Section 11.5, “Rebalancing Volumes”.

11.3. Shrinking Volumes

You can shrink volumes, as needed, while the cluster is online and available. For example, you might need to remove a brick that has become inaccessible in a distributed volume due to hardware or network failure.

Note

Data residing on the brick that you are removing will no longer be accessible at the Gluster mount point. Note however that only the configuration information is removed - you can continue to access the data directly from the brick, as necessary.
When shrinking distributed replicated and distributed striped volumes, you need to remove a number of bricks that is a multiple of the replica or stripe count. For example, to shrink a distributed striped volume with a stripe count of 2, you need to remove bricks in multiples of 2 (such as 4, 6, 8, etc.). In addition, the bricks you are trying to remove must be from the same sub-volume (the same replica or stripe set).
To shrink a volume
  1. Remove the brick using the following command:
    # gluster volume remove-brick VOLNAME BRICK
    For example, to remove server2:/exp2: 
    # gluster volume remove-brick test-volume server2:/exp2

Removing brick(s) can result in data loss. Do you want to Continue? (y/n)
  2. Enter "y" to confirm the operation. The command displays the following: 
    Remove Brick successful
  3. Check the volume information using the following command:
    # gluster volume info
    The command displays information similar to the following: 
    # gluster volume info
Volume Name: test-volume
Type: Distribute
Status: Started
Number of Bricks: 3
Bricks:
Brick1: server1:/exp1
Brick3: server3:/exp3
Brick4: server4:/exp4
  4. Rebalance the volume to ensure that all files are distributed to the new brick.
    You can use the rebalance command as described in Section 11.5, “Rebalancing Volumes”.

11.4. Migrating Volumes

You can migrate the data from one brick to another, as needed, while the cluster is online and available.
To migrate a volume
  1. Make sure the new brick, server5 in this example, is successfully added to the cluster.
    For more information, see Section 8.1, “Adding Servers to Trusted Storage Pool”.
  2. Migrate the data from one brick to another using the following command:
    # gluster volume replace-brick VOLNAME BRICKNEW-BRICK start
    For example, to migrate the data in server3:/exp3 to server5:/exp5 in test-volume: 
    # gluster volume replace-brick test-volume server3:/exp3  server5:exp5 start
Replace brick start operation successful

    Note

    You need to have the FUSE package installed on the server on which you are running the replace-brick command for the command to work.
  3. To pause the migration operation, if needed, use the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK pause
    For example, to pause the data migration from server3:/exp3 to server5:/exp5 in test-volume: 
    # gluster volume replace-brick test-volume server3:/exp3 server5:exp5 pause
Replace brick pause operation successful
  4. To abort the migration operation, if needed, use the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK abort
    For example, to abort the data migration from server3:/exp3 to server5:/exp5 in test-volume: 
    # gluster volume replace-brick test-volume server3:/exp3 server5:exp5 abort
Replace brick abort operation successful
  5. Check the status of the migration operation using the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK status
    For example, to check the data migration status from server3:/exp3 to server5:/exp5 in test-volume: 
    # gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 status
Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile 
Number of files migrated = 10567
Migration complete
    The status command shows the current file being migrated along with the current total number of files migrated. After completion of migration, it displays Migration complete.
  6. Commit the migration of data from one brick to another using the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK commit
    For example, to commit the data migration from server3:/exp3 to server5:/exp5 in test-volume: 
    # gluster volume replace-brick test-volume server3:/exp3 server5:/exp5 commit
replace-brick commit successful
  7. Verify the migration of brick by viewing the volume info using the following command:
    # gluster volume info VOLNAME
    For example, to check the volume information of new brick server5:/exp5 in test-volume: 
    # gluster volume info test-volume
Volume Name: testvolume
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/exp1
Brick2: server2:/exp2
Brick3: server4:/exp4
Brick4: server5:/exp5

The new volume details are displayed.
    The new volume details are displayed.
    In the above example, previously, there were bricks; 1,2,3, and 4 and now brick 3 is replaced by brick 5.

11.5. Rebalancing Volumes

After expanding or shrinking a volume (using the add-brick and remove-brick commands respectively), you need to rebalance the data among the servers. New directories created after expanding or shrinking of the volume will be evenly distributed automatically. For all the existing directories, the distribution can be fixed by rebalancing the layout and/or data. The rebalance operation has been split in to two phases for providing administrative control over when the actual data migration needs to occur – fixing the layout and migrating the data. If this is not necessary in your deployment, a simple rebalance will update the layout changes and migrate the data.
This section describes how to rebalance GlusterFS volumes in your storage environment, using the following common scenarios:
To view status of rebalance volume
You can display the status information about rebalance volume operation, as needed.
  • Check the status of the rebalance operation, using the following command:
    # gluster volume rebalance VOLNAME status
    For example: 
    # gluster volume rebalance test-volume status
Rebalance in progress: rebalanced 399 files of size 302047 (total files scanned 765)
    The time to complete the rebalance operation depends on the number of files on the volume along with the corresponding file sizes. Continue checking the rebalance status, verifying that the number of files rebalanced or total files scanned keeps increasing.
    For example, running the status command again might display a result similar to the following: 
    # gluster volume rebalance test-volume status
Rebalance completed: rebalanced 3107 files of size 1428745 (total files scanned 6000)
    The rebalance status displays the following when the rebalance is complete: 
    # gluster volume rebalance test-volume status
Rebalance completed!
To stop or pause the rebalance of a volume
  • Pause the rebalance operation, as necessary, using the following command:
    # gluster volume rebalance VOLNAME stop
    For example: 
    # gluster volume rebalance test-volume stop
Stopping rebalance on volume test-volume has been successful
    Restarting the rebalance operation causes the rebalance to continue from the point at which it was paused (stopped).

11.5.1. Rebalancing Volume to Fix Layout Changes

Fixing the layout is necessary because the layout structure is static for a given directory. In a scenario where new bricks have been added to the existing volume, newly created files in existing directories will still be distributed only among the old bricks. The # gluster volume rebalance VOLNAME fix-layout start command will fix the layout information so that the files can actually go to newly added nodes too. When this command is issued, all the file stat information which is already cached will get revalidated.
A fix-layout rebalance will only fix the layout changes and does not migrate data. If you want to migrate the existing data, use# gluster volume rebalance VOLNAME migrate-data start command to rebalance data among the servers.
To rebalance a volume to fix layout changes
  • Start the rebalance operation on any one of the server using the following command:
    # gluster volume rebalance VOLNAME fix-layout start
    For example: 
    # gluster volume rebalance test-volume fix-layout start
Starting rebalance on volume test-volume has been successful

11.5.2. Rebalancing Volume to Migrate Existing Data

When you add or remove a bricks to the existing to the volume, you need to rebalance the data among the servers for uniform distribution of data among the bricks. Now, issue # gluster volume rebalance VOLNAME migrate data start, to rebalance data among the servers. For effective data rebalancing, you should fix the layout first.
To rebalance a volume to migrate the existing data
  • Start the rebalance operation on any one of the server using the following command:
    # gluster volume rebalance VOLNAME migrate-data start
    For example: 
    # gluster volume rebalance test-volume migrate-data start
Starting rebalancing on volume test-volume has been successful

11.5.3. Rebalancing Volume to Fix Layout and Migrate Existing Data

After expanding or shrinking a volume (using the add-brick and remove-brick commands respectively), you need to rebalance the data among the servers.
To rebalance a volume to fix layout and migrate the existing data
  • Start the rebalance operation on any one of the server using the following command:
    # gluster volume rebalance VOLNAME start
    For example: 
    # gluster volume rebalance test-volume start
Starting rebalancing on volume test-volume has been successful

11.6. Stopping Volumes

To stop a volume
  1. Stop the volume using the following command:
    # gluster volume stop VOLNAME
    For example, to stop test-volume: 
    # gluster volume stop test-volume
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
  2. Enter y to confirm the operation. The output of the command displays the following: 
    Stopping volume test-volume has been successful

11.7. Deleting Volumes

To delete a volume
  1. Delete the volume using the following command:
    # gluster volume delete VOLNAME
    For example, to delete test-volume: 
    # gluster volume delete test-volume
Deleting volume will erase all information about the volume. Do you want to continue? (y/n)
  2. Enter y to confirm the operation. The command displays the following: 
    Deleting volume test-volume has been successful

11.8. Triggering Self-Heal on Replicate

Run the following command from FUSE mount point (on your clients):
# find gluster-mount -noleaf -print0 | xargs --null stat >/dev/null

Chapter 12. Managing GlusterFS Geo-replication

12.1. Replicated Volumes vs Geo-replication
12.2. Preparing to Deploy GlusterFS Geo-replication
12.2.1. Exploring Geo-replication Deployment Scenarios
12.2.2. GlusterFS Geo-replication Deployment Overview
12.2.3. Checking Geo-replication Minimum Requirement
12.2.4. Setting Up the Environment for Geo-replication
12.3. Starting GlusterFS Geo-replication
12.3.1. Starting Geo-replication
12.3.2. Verifying Successful Deployment
12.3.3. Displaying Geo-replication Status Information
12.3.4. Configuring Geo-replication
12.3.5. Stopping Geo-replication
12.4. Restoring Data from the Slave
12.5. Best Practices
GlusterFS Geo-replication provides a continuous, asynchronous, and incremental replication service from one site to another over Local Area Networks (LANs), Wide Area Network (WANs), and across the Internet.
GlusterFS Geo-replication provides a continuous, asynchronous, and incremental replication service from one site to another over Local Area Networks (LANs), Wide Area Network (WANs), and across the Internet.
GlusterFS Geo-replication uses a master–slave model, whereby replication and mirroring occurs between the following partners:
  • Master – A GlusterFS volume
  • Slave – A slave can be of the following types:
    • A local directory which can be represented as file URL like file:///path/to/dir. You can use shortened form, for example: /path/to/dir.
    • A GlusterFS Volume - Slave Volume can be either a local volume like gluster://localhost:volname (shortened form - :volname) or a volume served by different host like gluster://host:volname (shortened form - host:volname).

    Note

    Both of the above types can be accessed remotely using SSH tunnel. To use SSH, add an SSH prefix to either a file URL or gluster type URL. For example: ssh://root@remote-host:/path/to/dir (shortened form - root@remote-host:/path/to/dir) or ssh://root@remote-host:gluster://localhost:volname (shortened from - root@remote-host::volname).
This section introduces GlusterFS Geo-replication, illustrates the various deployment scenarios, and explains how to configure the system to provide replication and mirroring in your environment.

12.1. Replicated Volumes vs Geo-replication

The following table lists the difference between replicated volumes and geo-replication:
Replicated Volumes Geo-replication
Mirrors data across clusters Mirrors data across geographically distributed clusters
Provides high-availability Ensures backing up of data for disaster recovery
Synchronous replication (each and every file operation is sent across all the bricks) Asynchronous replication (checks for the changes in files periodically and syncs them on detecting differences)

12.2. Preparing to Deploy GlusterFS Geo-replication

This section provides an overview of the GlusterFS Geo-replication deployment scenarios, describes how you can check the minimum system requirements, and explores common deployment scenarios.

12.2.1. Exploring Geo-replication Deployment Scenarios

GlusterFS Geo-replication provides an incremental replication service over Local Area Networks (LANs), Wide Area Network (WANs), and across the Internet. This section illustrates the most common deployment scenarios for GlusterFS Geo-replication, including the following:
  • Geo-replication over LAN
  • Geo-replication over WAN
  • Geo-replication over the Internet
  • Multi-site cascading Geo-replication
Geo-replication over LAN
You can configure GlusterFS Geo-replication to mirror data over a Local Area Network.
Geo-replication over LAN
Geo-replication over WAN
You can configure GlusterFS Geo-replication to replicate data over a Wide Area Network.
Geo-replication over WAN
Geo-replication over Internet
You can configure GlusterFS Geo-replication to mirror data over the Internet.
Geo-replication over Internet
Multi-site cascading Geo-replication
You can configure GlusterFS Geo-replication to mirror data in a cascading fashion across multiple sites.
Multi-site cascading Geo-replication

12.2.2. GlusterFS Geo-replication Deployment Overview

Deploying GlusterFS Geo-replication involves the following steps:
  1. Verify that your environment matches the minimum system requirement. For more information, see Section 12.2.3, “Checking Geo-replication Minimum Requirement”.
  2. Determine the appropriate deployment scenario. For more information, see Section 12.2.1, “Exploring Geo-replication Deployment Scenarios”.
  3. Start Gluster Geo-replication on master and slave systems, as required. For more information, see Section 12.3, “Starting GlusterFS Geo-replication”.

12.2.3. Checking Geo-replication Minimum Requirement

Before deploying GlusterFS Geo-replication, ensure that Red Hat Storage Software Appliance is installed on both Master and Slave servers.

12.2.4. Setting Up the Environment for Geo-replication

Time Synchronization
  • On bricks of a geo-replication master volume, all the servers' time must be uniform. You are recommended to set up NTP service to keep the bricks sync in time and avoid out-of-time sync effect.
    For example: In a Replicated volume where brick1 of the master is at 12.20 hrs and brick 2 of the master is at 12.10 hrs with 10 minutes time lag, all the changes in brick2 between this period may go unnoticed during synchronization of files with Slave.
To setup Gluster Geo-replication for SSH
Password-less login has to be set up between the host machine (where geo-replication Start command will be issued) and the remote machine (where slave process should be launched through SSH).
  1. On the node where geo-replication sessions are to be set up, run the following command:
    # ssh-keygen -f /etc/glusterd/geo-replication/secret.pem
    Press Enter twice to avoid passphrase.
  2. On an SSH slave where you want to use the account georep-user for geo-replication slave purposes, add the content of /etc/glusterd/geo-replication/secret.pem.pub to ~georep-user/.ssh/authorized_keys file.

    Note

    Create ~georep-user/.ssh/authorized_keys file if it does not exist, so that only georep-user has permission to access the .ssh directory and its subdirectories. As of now, georep-user must be a superuser or an alias of it, but this restriction will be removed in a future release.
  3. Repeat the above steps on all the slaves.

12.3. Starting GlusterFS Geo-replication

This section describes how to configure and start Gluster Geo-replication in your storage environment, and verify that it is functioning correctly.

12.3.1. Starting Geo-replication

To start Gluster Geo-replication
  • Start geo-replication between the hosts using the following command:
    # gluster volume geo-replication MASTER SLAVE start
    For example: 
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir start
Starting geo-replication session between Volume1
example.com:/data/remote_dir has been successful

    Note

    You may need to configure the service before starting Gluster Geo-replication. For more information, see Section 12.3.4, “Configuring Geo-replication”.

12.3.2. Verifying Successful Deployment

You can use the gluster command to verify the status of Gluster Geo-replication in your environment.
To verify the status Gluster Geo-replication
  • Verify the status by issuing the following command on host:
    # gluster volume geo-replication MASTER SLAVE status
    For example:
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir status 
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 root@example.com:/data/remote_dir  Starting....

12.3.3. Displaying Geo-replication Status Information

You can display status information about a specific geo-replication master session, or a particular master-slave session, or all geo-replication sessions, as needed.
To display geo-replication status information
  • Display information of all geo-replication sessions using the following command: 
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 root@example.com:/data/remote_dir  Starting....
  • Display information of a particular master slave session using the following command:
    # gluster volume geo-replication MASTER SLAVE status
    For example, to display information of Volume1 and example.com:/data/remote_dir
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir status
    The status of the geo-replication between Volume1 and example.com:/data/remote_dir is displayed.
  • Display information of all geo-replication sessions belonging to a master
    # gluster volume geo-replication MASTER status
    For example, to display information of Volume1 
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir status

MASTER    SLAVE                            STATUS
______    ______________________________   ____________
Volume1 ssh://root@example.com:gluster://127.0.0.1:remove_volume  OK

Volume1 ssh://root@example.com:file:///data/remote_dir  OK
    The status of a session could be one of the following four:
  • Starting: This is the initial phase of the Geo-replication session; it remains in this state for a minute, to make sure no abnormalities are present.
  • OK: The geo-replication session is in a stable state.
  • Faulty: The geo-replication session has witnessed some abnormality and the situation has to be investigated further. For further information, see Chapter 16, Troubleshooting GlusterFS section.
  • Corrupt: The monitor thread which is monitoring the geo-replication session has died. This situation should not occur normally, if it persists contact Red Hat Supportwww.redhat.com/support/.

12.3.4. Configuring Geo-replication

To configure Gluster Geo-replication
  • Use the following command at the Gluster command line:
    # gluster volume geo-replication MASTER SLAVE config [options]
    For more information about the options, see Chapter 17, Command Reference .
    For example:
    To view list of all option/value pair, use the following command:
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir config

12.3.5. Stopping Geo-replication

You can use the gluster command to stop Gluster Geo-replication (syncing of data from Master to Slave) in your environment.
To stop Gluster Geo-replication
  • Stop geo-replication between the hosts using the following command:
    # gluster volume geo-replication MASTER SLAVE stop
    For example: 
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir stop
Stopping geo-replication session between Volume1 and
example.com:/data/remote_dir has been successful
    See Chapter 17, Command Reference for more information about the gluster command.

12.4. Restoring Data from the Slave

You can restore data from the slave to the master volume, whenever the master volume becomes faulty for reasons like hardware failure.
The example in this section assumes that you are using the Master Volume (Volume1) with the following configuration: 
machine1# gluster volume info
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: machine1:/export/dir16
Brick2: machine2:/export/dir16
Options Reconfigured:
geo-replication.indexing: on
The data is syncing from master volume (Volume1) to slave directory (example.com:/data/remote_dir). To view the status of this geo-replication session run the following command on Master: 
# gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status

MASTER    SLAVE                             STATUS
______    ______________________________    ____________
Volume1  root@example.com:/data/remote_dir   OK
Before Failure
Assume that the Master volume had 100 files and was mounted at /mnt/gluster on one of the client machines (client). Run the following command on Client machine to view the list of files: 
client# ls /mnt/gluster | wc –l
100
The slave directory (example.com) will have same data as in the master volume and same can be viewed by running the following command on slave: 
example.com# ls /data/remote_dir/ | wc –l
100
After Failure
If one of the bricks (machine2) fails, then the status of Geo-replication session is changed from "OK" to "Faulty". To view the status of this geo-replication session run the following command on Master: 
# gluster volume geo-replication Volume1 root@example.com:/data/remote_dir status

MASTER    SLAVE                              STATUS
______    ______________________________     ____________
Volume1   root@example.com:/data/remote_dir  Faulty
Machine2 is failed and now you can see discrepancy in number of files between master and slave. Few files will be missing from the master volume but they will be available only on slave as shown below.
Run the following command on Client: 
client # ls /mnt/gluster | wc –l
52
Run the following command on slave (example.com): 
Example.com# # ls /data/remote_dir/ | wc –l
100
To restore data from the slave machine
  1. Stop all Master's geo-replication sessions using the following command:
    # gluster volume geo-replication MASTER SLAVE stop
    For example: 
    machine1# gluster volume geo-replication Volume1
example.com:/data/remote_dir stop

Stopping geo-replication session between Volume1 &
example.com:/data/remote_dir has been successful

    Note

    Repeat # gluster volume geo-replication MASTER SLAVE stop command on all active geo-replication sessions of master volume.
  2. Replace the faulty brick in the master by using the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK start
    For example: 
    machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 start
Replace-brick started successfully
  3. Commit the migration of data using the following command:
    # gluster volume replace-brick VOLNAME BRICK NEW-BRICK commit force
    For example: 
    machine1# gluster volume replace-brick Volume1 machine2:/export/dir16 machine3:/export/dir16 commit force
Replace-brick commit successful
  4. Verify the migration of brick by viewing the volume info using the following command:
    # gluster volume info VOLNAME
    For example: 
    machine1# gluster volume info
Volume Name: Volume1
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: machine1:/export/dir16
Brick2: machine3:/export/dir16
Options Reconfigured:
geo-replication.indexing: on
  5. Run rsync command manually to sync data from slave to master volume's client (mount point).
    For example:
    example.com# rsync -PavhS --xattrs --ignore-existing /data/remote_dir/ client:/mnt/gluster
    Verify that the data is synced by using the following command:
    On master volume, run the following command: 
    Client # ls | wc –l
100
    On the Slave run the following command: 
    example.com# ls /data/remote_dir/ | wc –l
100
    Now Master volume and Slave directory is synced.
  6. Restart geo-replication session from master to slave using the following command:
    # gluster volume geo-replication MASTER SLAVE start
    For example: 
    machine1# gluster volume geo-replication Volume1
example.com:/data/remote_dir start
Starting geo-replication session between Volume1 &
example.com:/data/remote_dir has been successful

12.5. Best Practices

Manually Setting Time
If you have to change the time on your bricks manually, then you must set uniform time on all bricks. This avoids the out-of-time sync issue described in Section 12.2.4, “Setting Up the Environment for Geo-replication”. Setting time backward corrupts the geo-replication index, so the recommended way to set the time manually is:
  1. Stop geo-replication between the master and slave using the following command:
    # gluster volume geo-replication MASTER SLAVE stop
  2. Stop the geo-replication indexing using the following command:
    # gluster volume set MASTER geo-replication.indexing off
  3. Set uniform time on all bricks.s
  4. Restart your geo-replication sessions by using the following command:
    # gluster volume geo-replication MASTER SLAVE start
Running Geo-replication commands in one system
It is advisable to run the geo-replication commands in one of the bricks in the trusted storage pool. This is because, the log files for the geo-replication session would be stored in the *Server* where the Geo-replication start is initiated. Hence it would be easier to locate the log-files when required.
Isolation
Geo-replication slave operation is not sandboxed as of now and is ran as a privileged service. So for the security reason, it is advised to create a sandbox environment (dedicated machine / dedicated virtual machine / chroot/container type solution) by the administrator to run the geo-replication slave in it. Enhancement in this regard will be available in follow-up minor release.

Chapter 13. Monitoring your GlusterFS Workload

13.1. Running GlusterFS Volume Profile Command
13.1.1. Start Profiling
13.1.2. Displaying the I/0 Information
13.1.3. Stop Profiling
13.2. Running GlusterFS Volume TOP Command
13.2.1. Viewing Open fd Count and Maximum fd Count
13.2.2. Viewing Highest File Read Calls
13.2.3. Viewing Highest File Write Calls
13.2.4. Viewing Highest Open Calls on Directories
13.2.5. Viewing Highest Read Calls on Directory
13.2.6. Viewing List of Read Performance on each Brick
13.2.7. Viewing List of Write Performance on each Brick
Using GlusterFS Volume Top and Profile commands, you can monitor different parameters of the workload, thereby helping in capacity planning and performance tuning tasks of the GlusterFS volume.
You can view the performance and identify bottlenecks/hotspots of each brick of a volume. This helps system administrators to get vital performance information whenever performance needs to be probed.
This section describes how to run GlusterFS Volume Profile and Top Commands:

13.1. Running GlusterFS Volume Profile Command

GlusterFS Volume Profile command provides an interface to get the per-brick I/O information for each File Operation (FOP) of a volume. The per brick information helps in identifying bottlenecks in the storage system.
This section describes how to run GlusterFS Volume Profile command by performing the following operations:

13.1.1. Start Profiling

You must start the Profiling to view the File Operation information for each brick.
To start profiling:
  • Start profiling using the following command:
# gluster volume profile VOLNAME start
For example, to start profiling on test-volume: 
# gluster volume profile test-volume start
Profiling started on test-volume
When profiling on the volume is started, the following additional options are displayed in the Volume Info: 
diagnostics.count-fop-hits: on

diagnostics.latency-measurement: on

13.1.2. Displaying the I/0 Information

You can view the I/O information of each brick.
To display I/O information:
  • Display the I/O information using the following command:
# gluster volume profile VOLNAME info
For example, to see the I/O information on test-volume: 
# gluster volume profile test-volume info
Brick: Test:/export/2
Cumulative Stats:

Block                     1b+           32b+           64b+
Size:
       Read:                0              0              0
       Write:             908             28              8

Block                   128b+           256b+         512b+
Size:
       Read:                0               6             4
       Write:               5              23            16

Block                  1024b+          2048b+        4096b+
Size:
       Read:                 0              52           17
       Write:               15             120          846

Block                   8192b+         16384b+      32768b+
Size:
       Read:                52               8           34
       Write:              234             134          286

Block                                  65536b+     131072b+
Size:
       Read:                               118          622
       Write:                             1341          594


%-latency  Avg-      Min-       Max-       calls     Fop
          latency   Latency    Latency  
___________________________________________________________
4.82      1132.28   21.00      800970.00   4575    WRITE
5.70       156.47    9.00      665085.00   39163   READDIRP
11.35      315.02    9.00     1433947.00   38698   LOOKUP
11.88     1729.34   21.00     2569638.00    7382   FXATTROP
47.35   104235.02 2485.00     7789367.00     488   FSYNC

------------------

------------------

Duration     : 335

BytesRead    : 94505058

BytesWritten : 195571980

13.1.3. Stop Profiling

You can stop profiling the volume, if you do not need profiling information anymore.
To stop profiling
  • Stop profiling using the following command:
    # gluster volume profile VOLNAME stop
    For example, to stop profiling on test-volume:
    # gluster volume profile test-volume stop
    Profiling stopped on test-volume

13.2.  Running GlusterFS Volume TOP Command

GlusterFS Volume Top command allows you to view the glusterfs bricks’ performance metrics like read, write, file open calls, file read calls, file write calls, directory open calls, and directory real calls. The top command displays up to 100 results.
This section describes how to run and view the results for the following GlusterFS Top commands:

13.2.1. Viewing Open fd Count and Maximum fd Count

You can view both current open fd count (list of files that are currently the most opened and the count) on the brick and the maximum open fd count (count of files that are the currently open and the count of maximum number of files opened at any given point of time, since the servers are up and running). If the brick name is not specified, then open fd metrics of all the bricks belonging to the volume will be displayed.
To view open fd count and maximum fd count:
  • View open fd count and maximum fd count using the following command:
    # gluster volume top VOLNAME open [brick BRICK-NAME] [list-cnt cnt]
    For example, to view open fd count and maximum fd count on brick server:/export of test-volume and list top 10 open calls:
    # gluster volume top test-volume open brick server:/export/ list-cnt 10
    Brick: server:/export/dir1
    Current open fd's: 34 Max open fd's: 209 
                 ==========Open file stats========

open            file name
call count     

2               /clients/client0/~dmtmp/PARADOX/
                COURSES.DB

11              /clients/client0/~dmtmp/PARADOX/
                ENROLL.DB

11              /clients/client0/~dmtmp/PARADOX/
                STUDENTS.DB

10              /clients/client0/~dmtmp/PWRPNT/
                TIPS.PPT

10              /clients/client0/~dmtmp/PWRPNT/
                PCBENCHM.PPT

9               /clients/client7/~dmtmp/PARADOX/
                STUDENTS.DB

9               /clients/client1/~dmtmp/PARADOX/
                STUDENTS.DB

9               /clients/client2/~dmtmp/PARADOX/
                STUDENTS.DB

9               /clients/client0/~dmtmp/PARADOX/
                STUDENTS.DB

9               /clients/client8/~dmtmp/PARADOX/
                STUDENTS.DB

13.2.2. Viewing Highest File Read Calls

You can view highest read calls on each brick. If brick name is not specified, then by default, list of 100 files will be displayed.
To view highest file Read calls:
  • View highest file Read calls using the following command:
    # gluster volume top VOLNAME read [brick BRICK-NAME] [list-cnt cnt]
    For example, to view highest Read calls on brick server:/export of test-volume:
    # gluster volume top test-volume read brick server:/export list-cnt 10
    Brick: server:/export/dir1 
              ==========Read file stats========

read              filename
call count

116              /clients/client0/~dmtmp/SEED/LARGE.FIL

64               /clients/client0/~dmtmp/SEED/MEDIUM.FIL

54               /clients/client2/~dmtmp/SEED/LARGE.FIL

54               /clients/client6/~dmtmp/SEED/LARGE.FIL

54               /clients/client5/~dmtmp/SEED/LARGE.FIL

54               /clients/client0/~dmtmp/SEED/LARGE.FIL

54               /clients/client3/~dmtmp/SEED/LARGE.FIL

54               /clients/client4/~dmtmp/SEED/LARGE.FIL

54               /clients/client9/~dmtmp/SEED/LARGE.FIL

54               /clients/client8/~dmtmp/SEED/LARGE.FIL

13.2.3. Viewing Highest File Write Calls

You can view list of files which has highest file write calls on each brick. If brick name is not specified, then by default, list of 100 files will be displayed.
To view highest file Write calls:
  • View highest file Write calls using the following command:
    # gluster volume top VOLNAME write [brick BRICK-NAME] [list-cnt cnt]
    For example, to view highest Write calls on brick server:/export of test-volume:
    # gluster volume top test-volume write brick server:/export list-cnt 10
    Brick: server:/export/dir1 
                   ==========Write file stats========
write call count   filename

83                /clients/client0/~dmtmp/SEED/LARGE.FIL

59                /clients/client7/~dmtmp/SEED/LARGE.FIL

59                /clients/client1/~dmtmp/SEED/LARGE.FIL

59                /clients/client2/~dmtmp/SEED/LARGE.FIL

59                /clients/client0/~dmtmp/SEED/LARGE.FIL

59                /clients/client8/~dmtmp/SEED/LARGE.FIL

59                /clients/client5/~dmtmp/SEED/LARGE.FIL

59                /clients/client4/~dmtmp/SEED/LARGE.FIL

59                /clients/client6/~dmtmp/SEED/LARGE.FIL

59                /clients/client3/~dmtmp/SEED/LARGE.FIL

13.2.4. Viewing Highest Open Calls on Directories

You can view list of files which has highest open calls on directories of each brick. If brick name is not specified, then the metrics of all the bricks belonging to that volume will be displayed.
To view list of open calls on each directory
  • View list of open calls on each directory using the following command:
    # gluster volume top VOLNAME opendir [brick BRICK-NAME] [list-cnt cnt]
    For example, to view open calls on brick server:/export/ of test-volume:
    # gluster volume top test-volume opendir brick server:/export list-cnt 10
    Brick: server:/export/dir1 
             ==========Directory open stats========

Opendir count     directory name

1001              /clients/client0/~dmtmp

454               /clients/client8/~dmtmp

454               /clients/client2/~dmtmp
 
454               /clients/client6/~dmtmp

454               /clients/client5/~dmtmp

454               /clients/client9/~dmtmp

443               /clients/client0/~dmtmp/PARADOX

408               /clients/client1/~dmtmp

408               /clients/client7/~dmtmp

402               /clients/client4/~dmtmp

13.2.5. Viewing Highest Read Calls on Directory

You can view list of files which has highest directory read calls on each brick. If brick name is not specified, then the metrics of all the bricks belonging to that volume will be displayed.
To view list of highest directory read calls on each brick
  • View list of highest directory read calls on each brick using the following command:
    # gluster volume top VOLNAME readdir [brick BRICK-NAME] [list-cnt cnt]
    For example, to view highest directory read calls on brick server:/export of test-volume:
    # gluster volume top test-volume readdir brick server:/export list-cnt 10
    Brick: server:/export/dir1 
    ==========Directory readdirp stats========

readdirp count           directory name

1996                    /clients/client0/~dmtmp

1083                    /clients/client0/~dmtmp/PARADOX

904                     /clients/client8/~dmtmp

904                     /clients/client2/~dmtmp

904                     /clients/client6/~dmtmp

904                     /clients/client5/~dmtmp

904                     /clients/client9/~dmtmp

812                     /clients/client1/~dmtmp

812                     /clients/client7/~dmtmp

800                     /clients/client4/~dmtmp

13.2.6. Viewing List of Read Performance on each Brick

You can view the read throughput of files on each brick. If brick name is not specified, then the metrics of all the bricks belonging to that volume will be displayed. The output will be the read throughput. 
       ==========Read throughput file stats========

read         filename                         Time
through
put(MBp
s)
 
2570.00    /clients/client0/~dmtmp/PWRPNT/      -2011-01-31
           TRIDOTS.POT                      15:38:36.894610  
2570.00    /clients/client0/~dmtmp/PWRPNT/      -2011-01-31
           PCBENCHM.PPT                     15:38:39.815310                                                             
2383.00    /clients/client2/~dmtmp/SEED/        -2011-01-31
           MEDIUM.FIL                       15:52:53.631499
                                              
2340.00    /clients/client0/~dmtmp/SEED/        -2011-01-31
           MEDIUM.FIL                       15:38:36.926198
                                              
2299.00   /clients/client0/~dmtmp/SEED/         -2011-01-31
          LARGE.FIL                         15:38:36.930445
                                             
2259.00   /clients/client0/~dmtmp/PARADOX/      -2011-01-31
          COURSES.X04                       15:38:40.549919

2221.00   /clients/client0/~dmtmp/PARADOX/      -2011-01-31
          STUDENTS.VAL                      15:52:53.298766
                                              
2221.00   /clients/client3/~dmtmp/SEED/         -2011-01-31
          COURSES.DB                        15:39:11.776780

2184.00   /clients/client3/~dmtmp/SEED/         -2011-01-31
          MEDIUM.FIL                        15:39:10.251764
                                              
2184.00   /clients/client5/~dmtmp/WORD/         -2011-01-31
          BASEMACH.DOC                      15:39:09.336572
This command will initiate a dd for the specified count and block size and measures the corresponding throughput.
To view list of read performance on each brick
  • View list of read performance on each brick using the following command:
    # gluster volume top VOLNAME read-perf [bs blk-size count count] [brick BRICK-NAME] [list-cnt cnt]
    For example, to view read performance on brick server:/export/ of test-volume, 256 block size of count 1, and list count 10:
    # gluster volume top test-volume read-perf bs 256 count 1 brick server:/export/ list-cnt 10
    Brick: server:/export/dir1 256 bytes (256 B) copied, Throughput: 4.1 MB/s 
           ==========Read throughput file stats========

read         filename                         Time
through
put(MBp
s)

2912.00   /clients/client0/~dmtmp/PWRPNT/    -2011-01-31
           TRIDOTS.POT                   15:38:36.896486
                                           
2570.00   /clients/client0/~dmtmp/PWRPNT/    -2011-01-31
           PCBENCHM.PPT                  15:38:39.815310
                                           
2383.00   /clients/client2/~dmtmp/SEED/      -2011-01-31
           MEDIUM.FIL                    15:52:53.631499
                                           
2340.00   /clients/client0/~dmtmp/SEED/      -2011-01-31
           MEDIUM.FIL                    15:38:36.926198

2299.00   /clients/client0/~dmtmp/SEED/      -2011-01-31
           LARGE.FIL                     15:38:36.930445
                                                      
2259.00  /clients/client0/~dmtmp/PARADOX/    -2011-01-31
          COURSES.X04                    15:38:40.549919
                                           
2221.00  /clients/client9/~dmtmp/PARADOX/    -2011-01-31
          STUDENTS.VAL                   15:52:53.298766
                                           
2221.00  /clients/client8/~dmtmp/PARADOX/    -2011-01-31
         COURSES.DB                      15:39:11.776780
                                           
2184.00  /clients/client3/~dmtmp/SEED/       -2011-01-31
          MEDIUM.FIL                     15:39:10.251764
                                           
2184.00  /clients/client5/~dmtmp/WORD/       -2011-01-31
         BASEMACH.DOC                    15:39:09.336572

13.2.7. Viewing List of Write Performance on each Brick

You can view list of write throughput of files on each brick. If brick name is not specified, then the metrics of all the bricks belonging to that volume will be displayed. The output will be the write throughput.
This command will initiate a dd for the specified count and block size and measures the corresponding throughput. To view list of write performance on each brick:
  • View list of write performance on each brick using the following command:
    # gluster volume top VOLNAME write-perf [bs blk-size count count] [brick BRICK-NAME] [list-cnt cnt]
    For example, to view write performance on brick server:/export/ of test-volume, 256 block size of count 1, and list count 10:
    # gluster volume top test-volume write-perf bs 256 count 1 brick server:/export/ list-cnt 10
    Brick: server:/export/dir1
    256 bytes (256 B) copied, Throughput: 2.8 MB/s 
           ==========Write throughput file stats========

write                filename                 Time
throughput
(MBps)
 
1170.00    /clients/client0/~dmtmp/SEED/     -2011-01-31
           SMALL.FIL                     15:39:09.171494

1008.00    /clients/client6/~dmtmp/SEED/     -2011-01-31
           LARGE.FIL                      15:39:09.73189

949.00    /clients/client0/~dmtmp/SEED/      -2011-01-31
          MEDIUM.FIL                     15:38:36.927426

936.00   /clients/client0/~dmtmp/SEED/       -2011-01-31
         LARGE.FIL                        15:38:36.933177    
897.00   /clients/client5/~dmtmp/SEED/       -2011-01-31
         MEDIUM.FIL                       15:39:09.33628

897.00   /clients/client6/~dmtmp/SEED/       -2011-01-31
         MEDIUM.FIL                       15:39:09.27713

885.00   /clients/client0/~dmtmp/SEED/       -2011-01-31
          SMALL.FIL                      15:38:36.924271

528.00   /clients/client5/~dmtmp/SEED/       -2011-01-31
         LARGE.FIL                        15:39:09.81893

516.00   /clients/client6/~dmtmp/ACCESS/    -2011-01-31
         FASTENER.MDB                    15:39:01.797317

Chapter 14. Managing Directory Quota

14.1. Enabling Quota
14.2. Disabling Quota
14.3. Setting or Replacing Disk Limit
14.4. Displaying Disk Limit Information
14.5. Updating Memory Cache Size
14.6. Removing Disk Limit
Directory quotas in GlusterFS allow you to set limits on usage of disk space by directories or volumes. The storage administrators can control the disk space utilization at the directory and/or volume levels in GlusterFS by setting limits to allocatable disk space at any level in the volume and directory hierarchy. This is particularly useful in cloud deployments to facilitate utility billing model.

Note

For now, only Hard limit is supported. Here, the limit cannot be exceeded and attempts to use more disk space or inodes beyond the set limit will be denied.
System administrators can also monitor the resource utilization to limit the storage for the users depending on their role in the organization.
You can set the quota at the following levels:
  • Directory level – limits the usage at the directory level
  • Volume level – limits the usage at the volume level

Note

You can set the disk limit on the directory even if it is not created. The disk limit is enforced immediately after creating that directory. For more information on setting disk limit, see Section 14.3, “Setting or Replacing Disk Limit ”.

14.1. Enabling Quota

You must enable Quota to set disk limits.
To enable quota
  • Enable the quota using the following command:
    # gluster volume quota VOLNAME enable
    For example, to enable quota on test-volume: 
    # gluster volume quota test-volume enable
Quota is enabled on /test-volume

14.2. Disabling Quota

You can disable Quota, if needed.
To disable quota:
  • Disable the quota using the following command:
    # gluster volume quota VOLNAME disable
    For example, to disable quota translator on test-volume: 
    # gluster volume quota test-volume disable
Quota translator is disabled on /test-volume

14.3. Setting or Replacing Disk Limit

You can create new directories in your storage environment and set the disk limit or set disk limit for the existing directories. The directory name should be relative to the volume with the export directory/mount being treated as "/".
To set or replace disk limit
  • Set the disk limit using the following command:
    # gluster volume quota VOLNAME limit-usage /directorylimit-value
    For example, to set limit on data directory on test-volume where data is a directory under the export directory: 
    # gluster volume quota test-volume limit-usage /data 10GB
Usage limit has been set on /data

    Note

    In a multi-level directory hierarchy, the strictest disk limit will be considered for enforcement.

14.4. Displaying Disk Limit Information

You can display disk limit information on all the directories on which the limit is set.
To display disk limit information
  • Display disk limit information of all the directories on which limit is set, using the following command:
    # gluster volume quota VOLNAME list
    For example, to see the set disks limit on test-volume: 
    # gluster volume quota test-volume list

 Path__________Limit______Set Size 
/Test/data    10 GB       6 GB
/Test/data1   10 GB       4 GB
  • Display disk limit information on a particular directory on which limit is set, using the following command:
    # gluster volume quota VOLNAME list /directory name
    For example, to see the set limit on /data directory of test-volume: 
    # gluster volume quota test-volume list /data

Path__________Limit______Set Size
/Test/data    10 GB       6 GB

14.5.  Updating Memory Cache Size

For performance reasons, quota caches the directory sizes on client. You can set timeout indicating the maximum valid duration of directory sizes in cache, from the time they are populated.
For example: If there are multiple clients writing to a single directory, there are chances that some other client might write till the quota limit is exceeded. However, this new file-size may not get reflected in the client till size entry in cache has become stale because of timeout. If writes happen on this client during this duration, they are allowed even though they would lead to exceeding of quota-limits, since size in cache is not in sync with the actual size. When timeout happens, the size in cache is updated from servers and will be in sync and no further writes will be allowed. A timeout of zero will force fetching of directory sizes from server for every operation that modifies file data and will effectively disables directory size caching on client side.
To update the memory cache size
  • Update the memory cache size using the following command:
    # gluster volume set VOLNAME features.quota-timeout value
    For example, to update the memory cache size for every 5 seconds on test-volume: 
    # gluster volume set test-volume features.quota-timeout 5
Set volume successful

14.6. Removing Disk Limit

You can remove set disk limit, if you do not want quota anymore.
To remove disk limit
  • Remove disk limit set on a particular directory using the following command:
    # gluster volume quota VOLNAME remove /directory name
    For example, to remove the disk limit on /data directory of test-volume: 
    # gluster volume quota test-volume remove /data
Usage limit set on /data is removed

Chapter 15. POSIX Access Control Lists

15.1. Activating POSIX ACLs Support
15.1.1. Activating POSIX ACLs Support on Sever
15.1.2. Activating POSIX ACLs Support on Client
15.2. Setting POSIX ACLs
15.2.1. Setting Access ACLs
15.2.2. Setting Default ACLs
15.3. Retrieving POSIX ACLs
15.4. Removing POSIX ACLs
15.5. Samba and ACLs
15.6. NFS and ACLs
POSIX Access Control Lists (ACLs) allows you to assign different permissions for different users or groups even though they do not correspond to the original owner or the owning group.
For example: User john creates a file but does not want to allow anyone to do anything with this file, except another user, antony (even though there are other users that belong to the group john).
This means, in addition to the file owner, the file group, and others, additional users and groups can be granted or denied access by using POSIX ACLs.

Note

POSIX ACLs behavior varies with kernel versions.

15.1. Activating POSIX ACLs Support

To use POSIX ACLs for a file or directory, the partition of the file or directory must be mounted with POSIX ACLs support.

15.1.1. Activating POSIX ACLs Support on Sever

To mount the backend export directories for POSIX ACLs support, use the following command:
# mount -o acl device-namepartition
For example:
# mount -o acl /dev/sda1 /export1
Alternatively, if the partition is listed in the /etc/fstab file, add the following entry for the partition to include the POSIX ACLs option:
LABEL=/work /export1 ext3 rw, acl 14

15.1.2. Activating POSIX ACLs Support on Client

To mount the glusterfs volumes for POSIX ACLs support, use the following command:
# mount –t glusterfs -o acl severname:volume-idmount point
For example:
# mount -t glusterfs -o acl 198.192.198.234:glustervolume /mnt/gluster

15.2. Setting POSIX ACLs

You can set two types of POSIX ACLs, that is, access ACLs and default ACLs. You can use access ACLs to grant permission for a specific file or directory. You can use default ACLs only on a directory but if a file inside that directory does not have an ACLs, it inherits the permissions of the default ACLs of the directory.
You can set ACLs for per user, per group, for users not in the user group for the file, and via the effective right mask.

15.2.1. Setting Access ACLs

You can apply access ACLs to grant permission for both files and directories.
To set or modify Access ACLs
You can set or modify access ACLs use the following command:
# setfacl –m entry type file
The ACL entry types are the POSIX ACLs representations of owner, group, and other.
Permissions must be a combination of the characters r (read), w (write), and x (execute). You must specify the ACL entry in the following format and can specify multiple entry types separated by commas.
ACL Entry Description
u:uid:<permission> Sets the access ACLs for a user. You can specify user name or UID
g:gid:<permission> Sets the access ACLs for a group. You can specify group name or GID.
m:<permission> Sets the effective rights mask. The mask is the combination of all access permissions of the owning group and all of the user and group entries.
o:<permission> Sets the access ACLs for users other than the ones in the group for the file.
If a file or directory already has an POSIX ACLs, and the setfacl command is used, the additional permissions are added to the existing POSIX ACLs or the existing rule is modified.
For example, to give read and write permissions to user antony:
# setfacl -m u:antony:rw /mnt/gluster/data/testfile

15.2.2. Setting Default ACLs

You can apply default ACLs only to directories. They determine the permissions of a file system objects that inherits from its parent directory when it is created.
To set default ACLs
You can set default ACLs for files and directories using the following command:
# setfacl –m –-set entry type directory
For example, to set the default ACLs for the /data directory to read for users not in the user group:
# setfacl –m --set o::r /mnt/gluster/data

Note

An access ACLs set for an individual file can override the default ACLs permissions.
Effects of a Default ACLs
The following are the ways in which the permissions of a directory's default ACLs are passed to the files and subdirectories in it:
  • A subdirectory inherits the default ACLs of the parent directory both as its default ACLs and as an access ACLs.
  • A file inherits the default ACLs as its access ACLs.

15.3. Retrieving POSIX ACLs

You can view the existing POSIX ACLs for a file or directory.
To view existing POSIX ACLs
  • View the existing access ACLs of a file using the following command:
    # getfacl path/filename
    For example, to view the existing POSIX ACLs for sample.jpg 
    # getfacl /mnt/gluster/data/test/sample.jpg
# owner: antony
# group: antony
user::rw-
group::rw-
other::r--
  • View the default ACLs of a directory using the following command:
    # getfacl directory name
    For example, to view the existing ACLs for /data/doc 
    # getfacl /mnt/gluster/data/doc
# owner: antony
# group: antony
user::rw-
user:john:r--
group::r--
mask::r--
other::r--
default:user::rwx
default:user:antony:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

15.4. Removing POSIX ACLs

To remove all the permissions for a user, groups, or others, use the following command:
# setfacl -x ACL entry type file
For example, to remove all permissions from the user antony:
# setfacl -x u:antony /mnt/gluster/data/test-file

15.5. Samba and ACLs

If you are using Samba to access GlusterFS FUSE mount, then POSIX ACLs are enabled by default. Samba has been compiled with the --with-acl-support option, so no special flags are required when accessing or mounting a Samba share.

15.6. NFS and ACLs

Currently we do not support ACLs configuration through NFS, i.e. setfacl and getfacl commands do not work. However, ACLs permissions set using Gluster Native Client applies on NFS mounts.

Chapter 16. Troubleshooting GlusterFS

16.1. Managing GlusterFS Logs
16.1.1. Setting the Log Directory
16.1.2. Rotating Logs
16.2. Troubleshooting Geo-replication
16.2.1. Locating Log Files
16.2.2. Synchronization is not complete
16.2.3. Issues in Data Synchronization
16.2.4. Geo-replication status displays Faulty very often
16.2.5. Intermediate Master goes to Faulty State
16.3. Troubleshooting POSIX ACLs
16.3.1. setfacl command fails with “setfacl: <file or directory name>: Operation not supported” error
16.4. Troubleshooting NFS
16.4.1. mount command on NFS client fails with “RPC Error: Program not registered”
16.4.2. NFS server start-up fails with “Port is already in use” error in the log file."
16.4.3. mount command fails with “rpc.statd” related error message
16.4.4. mount command takes too long finish.
16.4.5. NFS server glusterfsd starts but initialization fails with “nfsrpc- service: portmap registration of program failed” error message in the log.
16.4.6. mount command fails with NFS server failed error.
16.4.7. showmount fails with clnt_create: RPC: Unable to receive
16.4.8. Application fails with "Invalid argument" or "Value too large for defined data type" error.
This section describes how to manage GlusterFS Logs and most common troubleshooting issues related to GlusterFS.

16.1. Managing GlusterFS Logs

This section describes how to manage GlusterFS logs by performing the following operations:
  • Setting the Log Directory
  • Rotating Logs

16.1.1. Setting the Log Directory

By default, all logs corresponding to bricks on the storage server are located in the /etc/glusterd/logs/bricks directory. You can set a different log directory for a volume, as needed.
To set the log directory for a volume
  1. Specify the log directory using the following command:
    # gluster volume log filename VOLNAME DIRECTORY
    For example, to set the log directory to /var/log/test-volume/ for test-volume: 
    # gluster volume log filename test-volume /var/log/test-volume/
log filename : successful

    Note

    Be sure that the directory exists before issuing the command.
  2. Optionally, verify the location of the log directory using the following command:
    # gluster volume log locate VOLNAME
    For example: 
    # gluster volume log locate test-volume
log file location: /var/log/test-volume

16.1.2. Rotating Logs

Administrators can rotate the log file in a volume, as needed.
To rotate a log file
  • Rotate the log file using the following command:
    # gluster volume log rotate VOLNAME
    For example, to rotate the log file on test-volume: 
    # gluster volume log rotate test-volume
log rotate successful

    Note

    When a log file is rotated, the contents of the current log file are moved to log-file- name.epoch-time-stamp.

16.2. Troubleshooting Geo-replication

This section describes the most common troubleshooting issues related to GlusterFS Geo-replication.

16.2.1. Locating Log Files

For every Geo-replication session, the following three log files are associated to it (four, if slave is a gluster volume):
  • Master-log-file - log file for the process which monitors the Master volume
  • Slave-log-file - log file for process which initiates the changes in slave
  • Master-gluster-log-file - log file for the maintenance mount point that Geo-replication module uses to monitor the master volume
  • Slave-gluster-log-file - is the slave's counterpart of it
Master Log File
To get the Master-log-file for geo-replication, use the following command:
gluster volume geo-replication MASTER SLAVE config log-file
For example:
# gluster volume geo-replication Volume1 example.com:/data/remote_dir config log-file
Slave Log File
To get the log file for Geo-replication on slave (glusterd must be running on slave machine), use the following commands:
  1. On master, run the following command:
    # gluster volume geo-replication Volume1 example.com:/data/remote_dir config session-owner 5f6e5200-756f-11e0-a1f0-0800200c9a66
    Displays the session owner details.
  2. On slave, run the following command:
    # gluster volume geo-replication /data/remote_dir config log-file /var/log/gluster/${session-owner}:remote-mirror.log
  3. Replace the session owner details (output of Step 1) to the output of the Step 2 to get the location of the log file.
    /var/log/gluster/5f6e5200-756f-11e0-a1f0-0800200c9a66:remote-mirror.log

16.2.2. Synchronization is not complete

Description: GlusterFS Geo-replication did not synchronize the data completely but still the geo- replication status display OK.
Solution: You can enforce a full sync of the data by erasing the index and restarting GlusterFS Geo- replication. After restarting, GlusterFS Geo-replication begins synchronizing all the data. All files will be compared using checksum, which can be a lengthy and high resource utilization operation on large data sets. If the error situation persists, contact Gluster Support.
For more information about erasing index, see Section 11.1, “Tuning Volume Options”.

16.2.3. Issues in Data Synchronization

Description: GlusterFS Geo-replication display status as OK, but the files do not get synced, only directories and symlink gets synced with error messages in log:
[2011-05-02 13:42:13.467644] E [master:288:regjob] GMaster: failed to sync ./some_file`
Solution: GlusterFS Geo-replication invokes rsync v3.07 in the host and the remote machine, check if you have the desired version installed.

16.2.4. Geo-replication status displays Faulty very often

Description: GlusterFS Geo-replication display status as faulty, very often with a backtrace similar to the following:
2011-04-28 14:06:18.378859] E [syncdutils:131:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/local/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 152, in twraptf(*aa) File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in listen rid, exc, res = recv(self.inf) File "/usr/local/libexec/glusterfs/python/syncdaemon/repce.py", line 42, in recv return pickle.load(inf) EOFError
Solution: This means that the RPC communication between the master gsyncd module and slave gsyncd module is broken and this can happen for various reasons. Check if it satisfies all the following pre-requisites:
  • Password-less SSH is set up properly between the host and the remote machine.
  • If FUSE is installed in the machine, since Geo-replication module mounts the GlusterFS volume using FUSE to sync data.
  • If the Slave is a volume, check if that volume is started.
  • If the Slave is a plain directory, check if the directory has been created already with the required permissions.
  • If GlusterFS 3.2 or higher is not installed in the default location (in Master) and has been prefixed to be installed in a custom location, configure the gluster-command for it to point to the exact location.
  • If GlusterFS 3.2 or higher is not installed in the default location (in slave) and has been prefixed to be installed in a custom location, configure the remote-gsyncd-command for it to point to the exact place where gsyncd is located.

16.2.5. Intermediate Master goes to Faulty State

Description: In a cascading set-up, the intermediate master goes to faulty state with the following log:
raise RuntimeError("aborting on uuid change from %s to %s" % \ RuntimeError: aborting on uuid change from af07e07c-427f-4586-ab9f- 4bf7d299be81 to de6b5040-8f4e-4575-8831-c4f55bd41154
Solution: In a cascading set-up the Intermediate master is loyal to the original primary master. The above log means that the GlusterFS Geo-replication module has detected change in primary master. If this is the desired behavior, delete the config option volume-id in the session initiated from the intermediate master.

16.3. Troubleshooting POSIX ACLs

This section describes the most common troubleshooting issues related to POSIX ACLs.

16.3.1. setfacl command fails with “setfacl: <file or directory name>: Operation not supported” error

This error is encountered when the backend file systems in one of the servers is not mounted with the "-o acl" option. The same can be confirmed by viewing the following error message in the log file of the server "Posix access control list is not supported".
Solution: Remount the backend file system with "-o acl" option. For more information, see Section 15.1.1, “Activating POSIX ACLs Support on Sever ”.

16.4. Troubleshooting NFS

This section describes the most common troubleshooting issues related to NFS .

16.4.1. mount command on NFS client fails with “RPC Error: Program not registered”

Start portmap or rpcbind service on the NFS server.
This error is encountered when the server has not started correctly.
On most Linux distributions this is fixed by starting portmap:
$ /etc/init.d/portmap start
On some distributions where portmap has been replaced by rpcbind, the following command is required:
$ /etc/init.d/rpcbind start
After starting portmap or rpcbind, gluster NFS server needs to be restarted.

16.4.2. NFS server start-up fails with “Port is already in use” error in the log file."

Another Gluster NFS server is running on the same machine.
This error can arise in case there is already a Gluster NFS server running on the same machine. This situation can be confirmed from the log file, if the following error lines exist:
[2010-05-26 23:40:49] E [rpc-socket.c:126:rpcsvc_socket_listen] rpc-socket: binding socket failed: Address already in use [2010-05-26 23:40:49] E [rpc-socket.c:129:rpcsvc_socket_listen] rpc-socket: Port is already in use [2010-05-26 23:40:49] E [rpcsvc.c:2636:rpcsvc_stage_program_register] rpc-service: could not create listening connection [2010-05-26 23:40:49] E [rpcsvc.c:2675:rpcsvc_program_register] rpc-service: stage registration of program failed [2010-05-26 23:40:49] E [rpcsvc.c:2695:rpcsvc_program_register] rpc-service: Program registration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465 [2010-05-26 23:40:49] E [nfs.c:125:nfs_init_versions] nfs: Program init failed [2010-05-26 23:40:49] C [nfs.c:531:notify] nfs: Failed to initialize protocols
To resolve this error one of the Gluster NFS servers will have to be shutdown. At this time, Gluster NFS server does not support running multiple NFS servers on the same machine.

16.4.3. mount command fails with “rpc.statd” related error message

If the mount command fails with the following error message:
mount.nfs: rpc.statd is not running but is required for remote locking. mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
Start rpc.statd
For NFS clients to mount the NFS server, rpc.statd service must be running on the clients.
Start rpc.statd service by running the following command:
$ rpc.statd

16.4.4. mount command takes too long finish.

Start rpcbind service on the NFS client.
The problem is that the rpcbind or portmap service is not running on the NFS client. The resolution for this is to start either of these services by running the following command:
$ /etc/init.d/portmap start
On some distributions where portmap has been replaced by rpcbind, the following command is required:
$ /etc/init.d/rpcbind start

16.4.5. NFS server glusterfsd starts but initialization fails with “nfsrpc- service: portmap registration of program failed” error message in the log.

NFS start-up can succeed but the initialization of the NFS service can still fail preventing clients from accessing the mount points. Such a situation can be confirmed from the following error messages in the log file:
[2010-05-26 23:33:47] E [rpcsvc.c:2598:rpcsvc_program_register_portmap] rpc-service: Could not register with portmap [2010-05-26 23:33:47] E [rpcsvc.c:2682:rpcsvc_program_register] rpc-service: portmap registration of program failed [2010-05-26 23:33:47] E [rpcsvc.c:2695:rpcsvc_program_register] rpc-service: Program registration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465 [2010-05-26 23:33:47] E [nfs.c:125:nfs_init_versions] nfs: Program init failed [2010-05-26 23:33:47] C [nfs.c:531:notify] nfs: Failed to initialize protocols [2010-05-26 23:33:49] E [rpcsvc.c:2614:rpcsvc_program_unregister_portmap] rpc-service: Could not unregister with portmap [2010-05-26 23:33:49] E [rpcsvc.c:2731:rpcsvc_program_unregister] rpc-service: portmap unregistration of program failed [2010-05-26 23:33:49] E [rpcsvc.c:2744:rpcsvc_program_unregister] rpc-service: Program unregistration failed: MOUNT3, Num: 100005, Ver: 3, Port: 38465
  1. Start portmap or rpcbind service on the NFS server.
    On most Linux distributions, portmap can be started using the following command:
    $ /etc/init.d/portmap start
    On some distributions where portmap has been replaced by rpcbind, run the following command:
    $ /etc/init.d/rpcbind start
    After starting portmap or rpcbind, gluster NFS server needs to be restarted.
  2. Stop another NFS server running on the same machine.
    Such an error is also seen when there is another NFS server running on the same machine but it is not the Gluster NFS server. On Linux systems, this could be the kernel NFS server. Resolution involves stopping the other NFS server or not running the Gluster NFS server on the machine. Before stopping the kernel NFS server, ensure that no critical service depends on access to that NFS server's exports.
    On Linux, kernel NFS servers can be stopped by using either of the following commands depending on the distribution in use:
    $ /etc/init.d/nfs-kernel-server stop
    $ /etc/init.d/nfs stop
  3. Restart Gluster NFS server.

16.4.6. mount command fails with NFS server failed error.

mount command fails with following error
mount: mount to NFS server '10.1.10.11' failed: timed out (retrying).
Perform one of the following to resolve this issue:
  1. Disable name lookup requests from NFS server to a DNS server.
    The NFS server attempts to authenticate NFS clients by performing a reverse DNS lookup to match hostnames in the volume file with the client IP addresses. There can be a situation where the NFS server either is not able to connect to the DNS server or the DNS server is taking too long to responsd to DNS request. These delays can result in delayed replies from the NFS server to the NFS client resulting in the timeout error seen above.
    NFS server provides a work-around that disables DNS requests, instead relying only on the client IP addresses for authentication. The following option can be added for successful mounting in such situations:
    option rpc-auth.addr.namelookup off

    Note

    Note: Remember that disabling the NFS server forces authentication of clients to use only IP addresses and if the authentication rules in the volume file use hostnames, those authentication rules will fail and disallow mounting for those clients.
    or
  2. NFS version used by the NFS client is other than version 3.
    Gluster NFS server supports version 3 of NFS protocol. In recent Linux kernels, the default NFS version has been changed from 3 to 4. It is possible that the client machine is unable to connect to the Gluster NFS server because it is using version 4 messages which are not understood by Gluster NFS server. The timeout can be resolved by forcing the NFS client to use version 3. The vers option to mount command is used for this purpose:
    $ mount nfsserver:export -o vers=3 mount-point

16.4.7. showmount fails with clnt_create: RPC: Unable to receive

Check your firewall setting to open ports 111 for portmap requests/replies and Gluster NFS server requests/replies. Gluster NFS server operates over the following port numbers: 38465, 38466, and 38467.

16.4.8. Application fails with "Invalid argument" or "Value too large for defined data type" error.

These two errors generally happen for 32-bit nfs clients or applications that do not support 64-bit inode numbers or large files. Use the following option from the CLI to make Gluster NFS return 32-bit inode numbers instead: nfs.enable-ino32 <on|off>
Applications that will benefit are those that were either:
  • built 32-bit and run on 32-bit machines such that they do not support large files by default
  • built 32-bit on 64-bit systems
This option is disabled by default so NFS returns 64-bit inode numbers by default.
Applications which can be rebuilt from source are recommended to rebuild using the following flag with gcc:
-D_FILE_OFFSET_BITS=64

Chapter 17. Command Reference

17.1. gluster Command
17.2. glusterd Daemon
This section describes the available commands and includes the following section:
  • gluster Command
    Gluster Console Manager (command line interpreter)
  • glusterd Daemon
    Gluster elastic volume management daemon

17.1. gluster Command

NAME
gluster - Gluster Console Manager (command line interpreter)
SYNOPSIS
To run the program and display the gluster prompt:
gluster
To specify a command directly: gluster [COMMANDS] [OPTIONS]
DESCRIPTION
The Gluster Console Manager is a command line utility for elastic volume management. You can run the gluster command on any export server. The command enables administrators to perform cloud operations such as creating, expanding, shrinking, rebalancing, and migrating volumes without needing to schedule server downtime.
COMMANDS
Command Description
Volume
volume info [all | VOLNAME] Displays information about all volumes, or the specified volume.
volume create NEW-VOLNAME [stripe COUNT] [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK ... Creates a new volume of the specified type using the specified bricks and transport type (the default transport type is tcp).
volume delete VOLNAME Deletes the specified volume.
volume start VOLNAME Starts the specified volume.
volume stop VOLNAME [force] Stops the specified volume.
volume rename VOLNAME NEW-VOLNAME Renames the specified volume.
volume help Displays help for the volume command.
Brick
volume add-brick VOLNAME NEW-BRICK ... Adds the specified brick to the specified volume.
volume replace-brick VOLNAME (BRICK NEW-BRICK) start | pause | abort | status Replaces the specified brick.
volume remove-brick VOLNAME [(replica COUNT)|(stripe COUNT)] BRICK ... Removes the specified brick from the specified volume.
Rebalance
volume rebalance VOLNAME start Starts rebalancing the specified volume.
volume rebalance VOLNAME stop Stops rebalancing the specified volume.
volume rebalance VOLNAME status Displays the rebalance status of the specified volume.
Log
volume log filename VOLNAME [BRICK] DIRECTORY Sets the log directory for the corresponding volume/brick.
volume log rotate VOLNAME [BRICK] Rotates the log file for corresponding volume/brick.
volume log locate VOLNAME [BRICK] Locates the log file for corresponding volume/brick.
Peer
peer probe HOSTNAME Probes the specified peer.
peer detach HOSTNAME Detachs the specified peer.
peer status Displays the status of peers.
peer help Displays help for the peer command.
Geo-replication
volume geo-replication MASTER SLAVE start
Start geo-replication between the hosts specified by MASTER and SLAVE. You can specify a local master volume as :VOLNAME.
You can specify a local slave volume as :VOLUME and a local slave directory as /DIRECTORY/SUB-DIRECTORY. You can specify a remote slave volume as DOMAIN::VOLNAME and a remote slave directory as DOMAIN:/DIRECTORY/SUB-DIRECTORY.
volume geo-replication MASTER SLAVE stop
Stop geo-replication between the hosts specified by MASTER and SLAVE. You can specify a local master volume as :VOLNAME and a local master directory as /DIRECTORY/SUB-DIRECTORY.
You can specify a local slave volume as :VOLNAME and a local slave directory as /DIRECTORY/SUB-DIRECTORY. You can specify a remote slave volume as DOMAIN::VOLNAME and a remote slave directory as DOMAIN:/DIRECTORY/SUB-DIRECTORY.
volume geo-replication MASTER SLAVE config [options] Configure geo-replication options between the hosts specified by MASTER and SLAVE.
gluster-command COMMAND The path where the gluster command is installed.
gluster-log-level LOGFILELEVEL The log level for gluster processes.
log-file LOGFILE The path to the geo-replication log file.
log-level LOGFILELEVEL The log level for geo-replication.
remote-gsyncd COMMAND The path where the gsyncd binary is installed on the remote machine.
ssh-command COMMAND The ssh command to use to connect to the remote machine (the default is ssh).
rsync-command COMMAND The rsync command to use for synchronizing the files (the default is rsync).
timeout SECONDS The timeout period.
sync-jobs N The number of simultaneous files/directories that can be synchronized.
volume_id= UID The command to delete the existing master UID for the intermediate/slave node.
Other
help Display the command options.
quit Exit the gluster command line interface.
FILES
/etc/glusterd/*
SEE ALSO
fusermount(1), mount.glusterfs(8), glusterfs-volgen(8), glusterfs(8), glusterd(8)

17.2. glusterd Daemon

NAME
glusterd - Gluster elastic volume management daemon
SYNOPSIS
glusterd [OPTION...]
DESCRIPTION
The glusterd daemon is used for elastic volume management. The daemon must be run on all export servers.
OPTIONS
Option Description
Basic
-l=LOGFILE, --log-file=LOGFILE Files to use for logging (the default is /usr/local/var/log/glusterfs/glusterfs.log).
-L=LOGLEVEL, --log-level=LOGLEVEL Logging severity. Valid options are TRACE, DEBUG, INFO, WARNING, ERROR and CRITICAL (the default is INFO).
--debug Runs the program in debug mode. This option sets --no-daemon, --log-level to DEBUG, and --log-file to console.
-N, --no-daemon Runs the program in the foreground.
Miscellaneous
-?, --help Displays this help.
--usage Displays a short usage message.
-V, --version Prints the program version.
FILES
/etc/glusterd/*
SEE ALSO
fusermount(1), mount.glusterfs(8), glusterfs-volgen(8), glusterfs(8), gluster(8)

Chapter 18. Glossary

Brick
A Brick is the GlusterFS basic unit of storage, represented by an export directory on a server in the trusted storage pool. A Brick is expressed by combining a server with an export directory in the following format:
SERVER:EXPORT
For example:
myhostname:/exports/myexportdir/
Cluster
A cluster is a group of linked computers, working together closely thus in many respects forming a single computer.
Distributed File System
A file system that allows multiple clients to concurrently access data over a computer network.
Filesystem
A method of storing and organizing computer files and their data. Essentially, it organizes these files into a database for the storage, organization, manipulation, and retrieval by the computer's operating system.
Source: Wikipedia
FUSE
Filesystem in Userspace (FUSE) is a loadable kernel module for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a "bridge" to the actual kernel interfaces.
Source: Wikipedia
Geo-Replication
Geo-replication provides a continuous, asynchronous, and incremental replication service from site to another over Local Area Networks (LAN), Wide Area Network (WAN), and across the Internet.
glusterd
The Gluster management daemon that needs to run on all servers in the trusted storage pool.
Metadata
Metadata is data providing information about one or more other pieces of data.
Namespace
Namespace is an abstract container or environment created to hold a logical grouping of unique identifiers or symbols. Each Gluster volume exposes a single namespace as a POSIX mount point that contains every file in the cluster.
Open Source
Open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology.
Before the term open source became widely adopted, developers and producers used a variety of phrases to describe the concept; open source gained hold with the rise of the Internet, and the attendant need for massive retooling of the computing source code.
Opening the source code enabled a self-enhancing diversity of production models, communication paths, and interactive communities. Subsequently, a new, three-word phrase "open source software" was born to describe the environment that the new copyright, licensing, domain, and consumer issues created.
Source: Wikipedia
Petabyte
A petabyte (derived from the SI prefix peta- ) is a unit of information equal to one quadrillion (short scale) bytes, or 1000 terabytes. The unit symbol for the petabyte is PB. The prefix peta- (P) indicates a power of 1000:
1 PB = 1,000,000,000,000,000 B = 10005 B = 1015 B.
The term "pebibyte" (PiB), using a binary prefix, is used for the corresponding power of 1024.
Source: Wikipedia
POSIX
Portable Operating System Interface (for Unix) is the name of a family of related standards specified by the IEEE to define the application programming interface (API), along with shell and utilities interfaces for software compatible with variants of the Unix operating system. Gluster exports a fully POSIX compliant file system.
RAID
Redundant Array of Inexpensive Disks (RAID) is a technology that provides increased storage reliability through redundancy, combining multiple low-cost, less-reliable disk drives components into a logical unit where all drives in the array are interdependent.
RRDNS
Round Robin Domain Name Service (RRDNS) is a method to distribute load across application servers. RRDNS is implemented by creating multiple A records with the same name and different IP addresses in the zone file of a DNS server.
Trusted Storage Pool
A storage pool is a trusted network of storage servers. When you start the first server, the storage pool consists of that server alone.
Userspace
Applications running in user space don’t directly interact with hardware, instead using the kernel to moderate access. Userspace applications are generally more portable than applications in kernel space. Gluster is a user space application.
Volfile
Volfile is a configuration file used by glusterfs process. Volfile will be usually located at /etc/glusterd/vols/VOLNAME.
Volume
A volume is a logical collection of bricks. Most of the gluster management operations happen on the volume.

Revision History

Revision History
Revision 1-8Mon Dec 20 2011Anjana Sriram
Updated System Requirements Chapter.
Revision 1-1Fri Nov 18 2011Daniel Macpherson
Transfer of book to Red Hat Documentation site