Dell PowerEdge SDS 100 Improving NFS performance on HPC clusters with Dell Flu
Dell PowerEdge SDS 100 Manual
View all Dell PowerEdge SDS 100 manuals
Add to My Manuals
Save this manual to your list of manuals |
Dell PowerEdge SDS 100 manual content summary:
- Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 1
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS This Dell technical white paper explains how to improve Network File System I/O performance by using Dell Fluid Cache for Direct Attached Storage in a High Performance Computing Cluster. Garima Kochhar Dell HPC Engineering March - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 2
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS This document is for informational purposes only and may contain typographical errors and technical inaccuracies. The content is provided as is, without express or implied warranties of any kind. © 2013 Dell Inc. All rights - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 3
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Contents Executive Summary ...5 1. Introduction...6 1.1. Dell Fluid Cache for DAS (direct-attached storage 6 2. Solution design and architecture 6 2.1. NFS storage solution (baseline 7 2.2. Dell Fluid Cache for DAS based - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 4
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Tables Table 1. Table 2. Table 3. Table 4. Table 5. NFS server and storage hardware configuration 8 NFS server software and firmware configuration 9 Hardware configuration for DFC 10 Software and firmware configuration for - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 5
performance improvements possible with DFC, especially for random I/O patterns. This white paper also includes a how-to recipe in the Appendix that provides step-by-step instructions on building the solution. 5 - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 6
Attached Storage) reduces this limitation by caching data while the backend storage services the I/O request, thus improving the performance of the entire NFS for DFC with NFS. Two appendices that provide step-by-step instructions on how to configure such a solution and provide information on - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 7
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS sections provide details on each of these components as well as information on tuning and monitoring the solution. 2.1. NFS storage solution (baseline) The baseline in this study is an NFS configuration. One PowerEdge R720 is - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 8
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Figure 2. NFS server Table 1. NFS server and storage hardware configuration Server configuration NFS SERVER PowerEdge R720 PROCESSORS Dual Intel(R) Xeon(R) CPU E5-2680 @ 2.70 GHz MEMORY 128 GB. 16 * 8 GB 1600MT/s RDIMMs - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 9
disks are combined using Linux Logical Volume Manager (LVM). DFC does not support caching of an LVM device, hence a single PERC was used for this summary of the design choices is provided in Section 2.4. Detailed instructions on configuring this storage solution are provided in Appendix A: Step-by - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 10
the same as in the baseline case. The I/O clients simply mount the same NFS exported directory as in the baseline configuration. Detailed instructions on configuring DFC for this storage solution are provided in Appendix A: Step-by-step configuration of Dell Fluid Cache for NFS. 2.3. I/O clients - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 11
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Table 5. I/O cluster details I/O cluster configuration CLIENTS CHASSIS CONFIGURATION INFINIBAND FABRIC For I/O traffic 64 PowerEdge M420 blade servers 32 blades in each of two PowerEdge M1000e chassis Two PowerEdge M1000e - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 12
was tuned based on experience gained from this study. This section provides a quick summary of some of the optimizations applied to the storage solution. Detailed instructions on configuring this storage solution are provided in Appendix A: Step-by-step configuration of Dell Fluid Cache for NFS. 12 - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 13
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS 2.4.1. Storage • 3TB NL SAS disks are selected for large capacity at a cost-effective price point. • Virtual disks are created using a RAID 60 layout. The RAID 6 span is across 10 data disks and 2 parity disks and the stripe is - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 14
. Dell recommends enabling the Logical Processor setting when using DFC. All results in this document have been achieved with this setting enabled. • DFC can support one to four SSDs in the solution. The DFC cache pool configuration used in this solution uses two PCIe SSDS. Two SSDs provide ~650GB - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 15
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS The warranty of the device is expressed in number of years and number of Petabytes written (PBW). For the recommended 350GB SSD drive, the standard warranty is 3 years, 25 PBW. The health of the device can be monitored using - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 16
that provides extensive statistics of the cache hits on the system, disk IO, etc. Additional details are available in the DFC User's Guide in [3]. 3. Performance This section presents the results of performance tests conducted on the storage solution described in Section 2. Performance tests were - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 17
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS • DFC in Write-Back mode (DFC-WB) - This configuration builds on the baseline by adding DFC as described in Section 2.2, and DFC is configured to operate in Write-Back (WB) mode. WB mode allows the caching of writes on the cache - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 18
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Throughput in MiB/s Figure 5. Large sequential write performance Sequential writes 2500 2000 1500 1000 500 0 1 2 4 8 16 32 48 64 Number of concurrent clients baseline DFC-WB DFC-WT Throughput in MiB/s Figure - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 19
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS better than the baseline since the data is already in the DFC cache. As expected on read operations, WB and WT tests have similar performance and can reach peak throughout of ~3050 MiB/s. 3.2. Random writes and reads Figure 7 - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 20
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS IOPS Figure 8. Random read performance 140000 120000 100000 80000 60000 40000 20000 0 1 Random reads 2 4 8 16 32 48 64 Number of concurrent clients baseline DFC-WB DFC-WT 3.3. Metadata tests This section presents - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 21
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Number of create() ops per second 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 Figure 9. Metadata file create performance File create 1 2 4 8 16 32 48 64 128 256 512 Number of concurrent clients baseline DFC - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 22
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Number of remove() ops per second 50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 Figure 11. Metadata file remove performance File remove 1 2 4 8 16 32 48 64 128 256 512 Number of concurrent clients - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 23
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Figure 12 shows that on a cold-cache read for sequential tests, the throughput of the DFC configuration drops from a peak of ~3,050 MiB/s to ~1,050 Mi/s. Data needs to be pulled from backend storage and hence the drop in - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 24
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Figure 13 shows that on a cold-cache read for the random tests, peak IOPS of the DFC configurations drop from ~123,000 IOPs to ~80,000 IOPs. Interestingly this is higher than the baseline IOPs of ~9,300, and explained below. - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 25
.com/downloads/global/products/pvaul/en/ssd_vs_hdd_price_and_performance_s tudy.pdf 3. Dell Fluid Cache for DAS Dell Fluid Cache for DAS User's Guide at www.dell.com/support http://www.dell.com/us/enterprise/p/poweredge-r720/pd 4. NFS tuning and the optimized Dell NFS Storage Solutions (NSS) Dell - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 26
-management/w/wiki/1760.openmanageserver-administrator-omsa.aspx 8. Dell PowerEdge Express Flash PCIe SSD www.dell.com/poweredge/expressflash http://support.dell.com/support/edocs/storage/Storlink/PCIe%20SSD/UG/en/index.htm http://content.dell.com/us/en/home/d/solutions/limited-hardware-warranties - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 27
with Dell Fluid Cache for DAS Appendix A: Step-by-step configuration of Dell Fluid Cache for NFS This appendix provides detailed step-by-step instructions on the configuration of the storage solution described in this white paper. Readers familiar with Dell's NSS line of solutions will find the - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 28
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Figure 15. Solution cabling A.2. NFS server set up After the PowerEdge R720 server is ready and cabled to the PowerVault MD1200, check Table 2 and Table 4 for details on the software used for the solution. 1. Create two virtual - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 29
(OMSA) v7.1.2 on the PowerEdge R720. Note that only v7.1.2 supports DFC at the time of writing. A newer OpenManage version cannot be as required. 8. Check that DFC is installed and the service is running. [root@nfs-dfc ~]# service fluid_cache status fldc_cfm (5794) is running... fldc_agent (5887) - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 30
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS 2. Change the OS I/O scheduler to "deadline". To the end of the kernel line in /etc/grub.conf for the .14.1 errata kernel, add elevator=deadline. 3. To work around a known error message with the PCIe SSDs, add the following - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 31
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Partitions Hot Spare Policy violated Encrypted Layout Size Associated Fluid Cache State Device Name Bus Protocol Media Read Policy Write Policy Cache Policy Stripe Element Size Disk Cache Policy : Available : Not Applicable : - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 32
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Status Name Slot ID State Firmware Version Minimum Required Firmware Version Driver Version Minimum Required Driver Version Storport Driver Version Minimum Required Storport Driver Version : Ok : PERC H810 Adapter - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 33
. Note that the share is exported with the 'sync' option. [root@nfs-dfc ~]# cat /etc/exports /home/xfs *(rw,no_root_squash,sync) 5. Restart the NFS service. Now the clients can mount the NFS share over the ib0 interface. Note that this solution recommends NFS v3 based on the results of a previous - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 34
and monitor DFC statistics. Check the fldcstat manual pages for options and descriptions of the statistics the Red Hat Storage Administration Guide. https://access.redhat.com/ implementing this alternative: [root@nfs-dfc ~]# service iptables stop; chkconfig iptables off A.7. Performance tuning - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 35
cache effects: • Unmount NFS share on clients. • Stop the NFS service and unmount the XFS file system on the server. • Mount XFS file system on the server and start the NFS service. • Mount NFS Share on clients. In addition for the cold cache tests - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 36
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS IOzone Argument -t +m -w -I -O Description Number of threads Location of clients to run IOzone on when in clustered mode Does not unlink (delete) temporary file Use O_DIRECT, bypass client cache Give results in ops/sec. For - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 37
IOzone random access patterns, the following procedure was followed to minimize cache effects during the metadata testing: • Unmount NFS share on clients. • Stop the NFS service and umount the XFS file system on the server. • Mount XFS file system on the server and start the NFS - Dell PowerEdge SDS 100 | Improving NFS performance on HPC clusters with Dell Flu - Page 38
Improving NFS Performance on HPC Clusters with Dell Fluid Cache for DAS Metadata file and directory creation test: # mpirun -np 32 -rr --hostfile ./hosts /nfs/share/mdtest -d /nfs/share/filedir -i 6 -b 320 -z 1 -L -I 3000 -y -u -t -C Metadata file and directory stat test: # mpirun -np 32 -rr --
Garima Kochhar
Dell HPC Engineering
March 2013, Version 1.0
Improving NFS Performance on HPC
Clusters with Dell Fluid Cache for DAS
This Dell technical white paper explains how to improve Network File
System I/O performance by using Dell Fluid Cache for Direct Attached
Storage in a High Performance Computing Cluster.