Dell DX6004S DX Object Storage Administration Guide - Page 60

Copyright © 2010 Caringo, Inc.

All rights reserved

55

Version 5.0

December 2010

differences, and any issues that could change the manner in which the DX Storage cluster will

process and store data.

If using USB boot devices, they may be removed from the running nodes in order to view and

backup the configuration and license files. The sticks or configuration server can then be updated

using the instructions in the README.txt file found in the ISO update cd for the new version of

DX Storage. After performing the upgrade, the node.cfg file should be validated to ensure there

are no deprecated parameters that need to be removed. See the release notes and Appendix A:

Node Configuration for information on any changes to the parameters in the node and/or cluster

configuration files.

Once all upgrades and validations have been completed, you can return USB devices to the same

node from which it was removed. Care should be taken to match each USB device to its original

node in the cluster to ensure the “vols” parameter which defines the storage devices matches the

correct node.

Before any node upgrade is performed, the administrator should verify the cluster’s health by

checking for critical error messages on each node’s status page or the SNMP CastorErrTable OID.

This is necessary in order to ensure that there are no hardware problems that could interrupt the

upgrade process. Any problems should be corrected prior to upgrading.

C.2.2. Upgrade Steps

1. Shutdown all cluster nodes (or one at a time for a rolling upgrade)

2. Install updated USB boot devices or updated ISO on your PXE boot server

3. Reboot all nodes

4. Verify correct operation

A simultaneous shutdown of the cluster is the first step in the simple upgrade. For customers where

downtime cannot be tolerated, the nodes can be rebooted one at a time in a rolling upgrade so

the cluster remains online. During a rolling upgrade, the absence of the single upgrading node

will be detected and the remaining nodes will begin trying to recover the missing content. The

recovery will be halted when the upgraded node comes back online. Best practice would be to pre-

prepare upgraded sticks so the downtime for the upgrading node is minimal and there is no risk of

the remaining nodes filling their disks with recovered content. If all the nodes are shutdown within

several seconds of each other, initiation of the disk recovery process is not a concern.

C.2.2.1. Example Shutdown Script

This Unix shell script demonstrates a method of issuing the shutdown command to all cluster nodes.

In this example, all the nodes of the cluster are defined in the NODES variable.

NODES=”192.168.1.101

192.168.1.102

192.168.1.103”

for n in $NODES; do

snmpset –v 1 –c pwd –m +CASTOR-MIB $n \

caringo.castor.CastorShutdownAction = ”shutdown”

done

This Unix shell script demonstrates a method of issuing the shutdown command to all cluster nodes.

In this example, all the nodes of the cluster are defined in the NODES variable.

C.2.2.2. Cluster Reboot

After the cluster has been shutdown, ensure the updated USB sticks or the configuration server

is prepared and then begin the reboot process. The recommended power-on sequence is to start

Section	Page
DX Object Storage Administration Guide	1
Table of Contents	3
Chapter 1. Welcome to DX Storage	6
1.1. Overview of DX Storage	6
1.2. Components	6
1.3. About this Document	6
1.3.1. Audience	6
1.3.2. Scope	6
Chapter 2. Introduction to the Admin Console	7
2.1. Accessing the Admin Console	7
2.2. Initial View of the Admin Console	7
2.2.1. Viewing the Cluster Status Page	7
2.2.2. Viewing a Node's Status Page	8
2.3. Printing the Admin Console	8
2.4. Viewing License Information	8
Chapter 3. Managing the Cluster	10
3.1. Viewing the Cluster Status Page	10
3.2. Authenticating Cluster-Wide Actions	11
3.3. Shutting Down or Restarting the Cluster	12
3.4. Viewing Nodes in the Cluster	12
3.5. Searching for Nodes By IP Address	13
3.6. Searching for Nodes by Status	13
3.7. Choosing and Preserving Cluster Settings	13
3.7.1. Enabling Logging	15
3.7.2. Replication	15
3.7.3. Suspend	15
3.7.4. Power	15
3.7.5. Managing Tenants	15
3.8. Cluster Name	15
3.9. Cluster Multicast Address	15
3.10. % Used Indicator	15
Chapter 4. Managing Tenants	17
4.1. Terminology Related to Tenant Security	18
4.2. About the Default Cluster Domain	19
4.3. Security Privileges for Administrative Operations	20
4.4. Rules and Recommendations for Managing Tenants	21
4.5. Domain Naming Rules	21
4.6. Adding, Editing, or Deleting Tenants	22
4.7. Other Cluster Administrator Tasks	24
4.7.1. Using Administrative Override	25
4.7.1.1. Using Override to Delete an Object	25
4.7.1.2. Using Override to GET or APPEND User Lists	25
4.7.1.3. Using Override to Resolve Authorization Specification Issues	27
4.7.2. Working With Inaccessible Objects	28
Chapter 5. Managing Nodes	30
5.1. Viewing the Node Status Page	30
5.1.1. Shutting Down or Restarting a Node	30
5.1.2. Retiring or Identifying a Node	30
5.1.2.1. Retiring a Node or Volume	30
5.1.2.2. Identifying a Volume	31
5.1.3. Errors and Announcements	31
5.1.4. Additional Node Status Information	31
5.1.4.1. Hardware Status Reporting	33
5.2. Displaying Subcluster Information	34
Chapter 6. Configuring the Node	35
6.1. Option Names and Descriptions	35
6.2. Managing DX Storage Administrators and Users	41
6.2.1. Defining CAStor administrators and SNMP Administrators	41
6.2.2. Defining DX Storage Operators	42
6.2.3. Securing the Administrator and Operator Passwords	42
6.3. Managing Content Integrity Settings	43
6.3.1. autoRepOnWrite	43
6.3.2. repPriority	43
6.3.3. autoValidateRead	43
6.4. Managing Other Stream Replication Settings	44
6.4.1. minreps, maxreps, and defreps	44
6.4.2. hpStartDelay	44
6.5. Managing Volumes	44
6.5.1. device	45
6.5.2. policy	46
6.5.3. Specifying Exceptions	46
6.6. Configuring Power Management Settings	46
6.6.1. sleepAfter	46
6.6.2. wakeAfter	46
6.7. Managing Other Settings	46
6.7.1. consolePort	46
6.7.2. domainHeaders	46
6.7.3. loghost	47
6.7.4. timeSource	47
Chapter 7. Managing Volumes	49
7.1. Volume Expiration	49
7.2. Movement Between Nodes	49
7.3. Physical Errors	49
Appendix A. Implementation of Multi-Server Chassis	51
A.1. Configuration Parameters	51
A.1.1. Processes parameter	51
A.1.2. Network Setup parameters	51
A.1.3. Using the vols parameter	52
A.1.4. Using the subcluster parameter	52
A.2. Monitoring and Administration	53
Appendix B. Using SNMP with DX Storage	54
B.1. SNMP Management Information Base (MIB) Reference	54
B.2. Managing DX Storage Nodes	54
B.2.1. Shutdown Action for Nodes	54
B.2.2. Retire Action for Nodes and Volumes	54
B.2.2.1. Single Volumes	55
B.2.2.2. Entire Node	55
B.3. SNMP Tools and Monitoring Systems	55
B.3.1. Open Source Tools	55
B.3.2. SNMP Examples with DX Storage	56
B.3.3. SNMP Action OIDs	56
B.3.3.1. castorShutdownAction	56
B.3.3.2. castorRetireAction	56
B.3.3.3. castorLogLevelAction	56
B.3.3.4. castorSyslogHostAction	56
B.3.3.5. volumeRecoverySuspend	57
B.3.4. Practical SNMP with DX Storage	57
B.3.4.1. Health Monitoring	57
B.3.4.2. Capacity Monitoring	57
B.3.4.3. Client Activity Reporting	58
Appendix C. Upgrading a License or Cluster	59
C.1. Upgrading a License File	59
C.2. Software Upgrade Overview	59
C.2.1. Preparation	59
C.2.2. Upgrade Steps	60
C.2.2.1. Example Shutdown Script	60
C.2.2.2. Cluster Reboot	60
C.3. Back-out Steps	61
Appendix D. Troubleshooting	62
D.1. Restoring Domains and Buckets	62
D.1.1. Recovering a Deleted Domain	62
D.1.2. Recovering a Deleted Bucket	65
D.2. Resolving Duplicate Domain Names in a Mirrored or Disaster Recovery (DR) Cluster	66
D.2.1. Renaming a Domain in its Source Cluster (DR Cluster Conflict Only)	66
D.2.2. Renaming a Domain in a Mirrored or DR Cluster	67
D.3. Using DX Content Router to List Buckets and Objects	69
D.4. Boot Errors	69
D.5. Configuration	70
D.6. Operational Problems	70
Appendix E. Drive Identification API	72
E.1. Overview	72
E.2. Customization Steps	72
Appendix F. Customizing the Admin Console	73

Dell DX6004S DX Object Storage Administration Guide - Page 60

C.2.2. Upgrade Steps, C.2.2.1. Example Shutdown Script, C.2.2.2. Cluster Reboot

Page 60 highlights