Home » Dell Manuals » Servers » Dell DX6012S » Manual Viewer

Dell DX6012S DX Object Storage Application Guide - Page 76

Basic Content Integrity Assurance

Add to My Manuals
Save this manual to your list of manuals

Page 76 highlights

Chapter 20. Basic Content Integrity Assurance DX Storage provides some basic capabilities for allowing applications to obtain and validate integrity guarantees on the stored data. Integrity here means an independently verifiable guarantee that the data returned for a given name or UUID is exactly the same data that was stored using that name or UUID, perhaps many months or years in the past. This is done by hashing the data using a cryptographic hash algorithm; content metadata is not included in the hash. If the application stores the name or UUID and its associated hash value, these can be used later to verify the content has not changed, either through accidental or malicious means. 20.1. Integrity Seals An integrity seal is a URL containing an object's name or UUID, its hash value, and the type of hash algorithm that was used for the computations. An application can request an integrity seal when it performs a WRITE by including a hashtype query string as follows: POST http://company.cluster.com/?hashtype=md5 HTTP/1.1 Currently allowable hashtypes are md5, sha1, sha256, sha384, and sha512. After creating the object and assigning a name or UUID, DX Storage replies with a 201 (Created) response that includes a location header with a URL that can later be used to retrieve the data. In addition to the host and name or UUID, the URL includes the hash type and hash value computed from the content object. This URL, including the triple name or UUID, hash type, and hash, is referred to as the integrity seal of the content object. Following is an example of a complete integrity seal embedded in a Location header for an unnamed object. Location: http://129.69.251.143/41A140B5271DC8D22FF8D027176A0821? hashtype=md5&hash=7A25E6067904EAC8002498CF1AE33023 20.2. Validating Reads An integrity seal can be used in a subsequent READ request to validate the data stored in a DX Storage cluster (any cluster). By supplying the URL returned in the Location header from the WRITE request (perhaps replacing the host address if connected to a different cluster or node), the application can ask DX Storage to validate while reading the data. For example: GET http://129.69.251.143/41A140B5271DC8D22FF8D027176A0821?hashtype=md5& hash=7A25E6067904EAC8002498CF1AE33023 HTTP/1.1 Upon receiving such a READ request, DX Storage recomputes the hash of the stored content using the supplied hashtype and compares the computed hash with the hash value in the integrity seal. If the two values do not match, DX Storage send a 200 (OK) response but drops the connection prior to sending the object content. If, on the other hand, the content has not been modified or corrupted in any way, the hashes match and, in that case, DX Storage returns the object as usual with the computed digest as a trailing Location header. Because the hash algorithms are published and well-known, it is also possible for users and third parties to independently validate an object that has been stored by DX Storage. This can be done by reading its contents, computing the hash value, and comparing it with the hash value in the seal. By publishing an integrity seal at the time it is created one can even provide indisputable evidence Copyright © 2010 Caringo, Inc. All rights reserved 71 Version 5.0 December 2010

Section	Page
DX Object Storage Application Guide	1
Table of Contents	3
Chapter 1. Introduction to DX Storage Objects	6
1.1. Introducing Named Objects	6
1.2. Summary of Named Objects and Unnamed Objects	7
1.3. Common Named Object Terminology	7
1.4. About DX Storage Object Types	8
1.5. Bucket and Object Naming Rules	8
1.6. Simple Examples of Creating Buckets and Named Objects	8
1.7. Universal Resource Identifiers (URIs)	9
1.8. About Objects and Security	10
1.8.1. The Basics of DX Storage Security	10
1.8.2. Commonly Used Security Terminology	11
1.8.3. Basic Security Guidelines	11
1.9. Concepts for Unnamed Objects	11
1.9.1. Universally Unique Identifiers (UUIDs)	11
1.9.2. About Immutable Objects	12
1.9.3. About Anchor Streams	12
1.10. Document Typographical Conventions	13
Chapter 2. Connecting to a DX Storage Cluster	14
2.1. About the Primary Access Node (PAN)	14
2.2. Requirements for Node Accessibility	14
2.2.1. Using the DX Storage Software Development Kit (SDK)	14
2.2.2. Using Multicast-DNS (mDNS)	15
2.2.3. Using DNS Round Robin	15
2.2.4. Using a Pool of Static IP Addresses	16
2.2.5. Using a Single Static IP Address	16
Chapter 3. Application Best Practices	17
3.1. Using the Software Development Kit (SDK)	17
3.2. Dynamically Locating DX Storage Nodes	17
3.3. Optimizing Bucket and Domain Operations	17
3.4. Using HTTP Client Libraries	17
3.5. Handling Redirects	18
3.6. Using Range Headers	18
3.7. Multi-Threading	19
3.8. Using Persistent Connections	19
Chapter 4. Introduction to the Simple Content Storage Protocol (SCSP)	20
4.1. Mapping SCSP Operations to HTTP Methods	20
4.2. HTTP Overview	21
4.3. Requests and Responses	21
4.3.1. HTTP Response Codes	21
4.3.2. Redirect Responses	22
4.4. Formatting SCSP Commands for Named and Unnamed Objects	22
4.5. Undefined Responses from DX Storage	23
4.6. Normal Response Headers	24
Chapter 5. SCSP READ	27
5.1. Introduction to READ	27
5.1.1. Reading With Content Validation	27
5.1.2. Requestion Node Status and Cluster Capacity	27
5.2. Unnamed Object READ Details	28
5.3. Normal Responses to READ	28
5.3.1. Normal READ Responses for Named Objects	28
5.3.1.1. Normal READ Response for a Domain	28
5.3.1.2. Normal READ Response for a Bucket	29
5.3.1.3. Normal READ Response for a Named Object	29
5.3.2. Normal READ Response for Unnamed Objects	29
5.3.3. Other Normal READ Responses	30
5.4. Error Responses to READ	30
Chapter 6. SCSP WRITE	32
6.1. Introduction to WRITE	32
6.1.1. About the Expect: 100-continue Header and WRITE	33
6.1.2. About Replicate On Write	33
6.2. Named Object WRITE Details	33
6.3. Unnamed Object WRITE Details	33
6.4. Normal Responses to WRITE	33
6.5. Error Responses to WRITE	34
Chapter 7. SCSP DELETE	36
7.1. Introduction to DELETE	36
7.2. Unnamed Object DELETE Details	37
7.3. Named Object DELETE Details	37
7.4. Normal Responses to DELETE	37
7.5. Error Responses to DELETE	38
Chapter 8. SCSP UPDATE	39
8.1. Introduction to UPDATE	39
8.2. Unnamed Object UPDATE Details	39
8.3. Normal Responses to UPDATE	40
Chapter 9. SCSP COPY	41
9.1. Introduction to COPY	41
9.2. Unnamed Object COPY Details	41
9.3. Normal Responses to COPY	41
Chapter 10. SCSP APPEND	42
10.1. Introduction to APPEND	42
10.2. Unnamed Object APPEND Details	43
10.3. Normal Responses to APPEND	43
Chapter 11. SCSP INFO	44
11.1. Introduction to INFO	44
11.1.1. Getting Replica Counts	44
11.1.2. Getting Node Status and Cluster Capacity	44
11.2. Unnamed Object INFO Details	45
11.3. Normal Responses to INFO	45
11.3.1. Normal INFO Responses for Named and Unnamed Objects	45
11.3.2. Normal INFO Responses for Named Objects	46
11.3.3. Normal INFO Responses for Unnamed Objects	47
11.4. Error Responses to INFO	47
Chapter 12. Introduction to Object Security	48
12.1. About Security Roles	48
12.2. Security Realm Overview	49
12.2.1. Common Security Terminology	49
12.2.2. About Security Realms	49
12.2.3. About Realm Caching and Security	49
12.3. About Authorization Header Syntax	50
12.3.1. About Realm Names	51
12.3.1.1. About Realms and Buckets	51
12.3.1.2. About owner@ and @owner Syntax	51
12.3.1.3. About user@realm Syntax	51
12.4. Authorization Specification Examples	51
12.5. About Authorization Header Evaluation	52
12.6. Creating Realms	53
12.7. Updating the Realm on a Bucket	54
12.8. Administrative Override and Security	54
Chapter 13. Managing Security for Application Developers	55
13.1. What Application Developers Need to Know	55
13.2. About Bucket Authorization	55
13.3. Example: Creating Bucket Owners	56
13.4. Examples of Creating Buckets and Named Objects	56
Chapter 14. Managing Security for Domain Managers	59
14.1. What Domain Managers Need to Know	59
14.2. About Domain Manager Tasks and Responsibilities	59
14.3. Examples of Updating Domain and Administrator User Lists	59
14.3.1. Updating the Domain Realm	60
14.3.2. Updating the Domain Manager Realm	61
14.3.3. Updating a Realm Using PUT	62
Chapter 15. Using the Content-MD5 Metadata Header	64
Chapter 16. Using Lifepoint Metadata Headers	65
16.1. About Storage Policies	65
16.2. A Simple Lifecycle Example	65
16.3. Lifepoint/Lifecycle Specification	66
16.4. Constraint Types	66
16.4.1. Replication	67
16.4.2. Deletion	67
Chapter 17. Using the Allow Metadata Header	69
17.1. Overview of the Allow Header	69
17.2. Administrative Override	69
Chapter 18. Using Caching Metadata Headers	71
18.1. About Caching Headers	71
18.2. Using HTTP/1.0 Caching Headers	71
18.2.1. Last-Modified	71
18.2.2. If-Modified-Since	71
18.2.3. If-Unmodified-Since	71
18.2.4. Expires	72
18.3. Using HTTP/1.1 Caching Headers	72
18.3.1. ETag	72
18.3.2. If-Match	72
18.3.3. If-None-Match	73
18.3.4. If-Range	73
Chapter 19. Using Custom Metadata Headers	75
Chapter 20. Basic Content Integrity Assurance	76
20.1. Integrity Seals	76
20.2. Validating Reads	76
20.3. Application Initiated Hash Upgrading	77
Appendix A. Content Header Metadata	78
Appendix B. Sample Code and Client Library Behaviors with 100-Continue	80

Match case Limit results 1 per page

Version 5.0

December 2010

Chapter 20. Basic Content Integrity Assurance

DX Storage provides some basic capabilities for allowing applications to obtain and validate integrity

guarantees on the stored data.

Integrity

here means an independently verifiable guarantee that

the data returned for a given name or UUID is exactly the same data that was stored using that

name or UUID, perhaps many months or years in the past. This is done by

hashing

the data using a

cryptographic hash algorithm; content metadata is not included in the hash. If the application stores

the name or UUID and its associated hash value, these can be used later to verify the content has

not changed, either through accidental or malicious means.

20.1. Integrity Seals

integrity seal

is a URL containing an object’s name or UUID, its hash value, and the type of hash

algorithm that was used for the computations. An application can request an integrity seal when it

performs a WRITE by including a

hashtype

query string as follows:

POST http://company.cluster.com/?hashtype=md5 HTTP/1.1

Currently allowable hashtypes are

md5

sha1

sha256

sha384

, and

sha512

. After creating the

object and assigning a name or UUID, DX Storage replies with a 201 (Created) response that

includes a location header with a URL that can later be used to retrieve the data.

In addition to the host and name or UUID, the URL includes the hash type and hash value computed

from the content object. This URL, including the triple name or UUID, hash type, and hash, is

referred to as the

integrity seal

of the content object.

Following is an example of a complete integrity seal embedded in a Location header for an

unnamed object.

Location: http://129.69.251.143/41A140B5271DC8D22FF8D027176A0821?

hashtype=md5&hash=7A25E6067904EAC8002498CF1AE33023

20.2. Validating Reads

An integrity seal can be used in a subsequent READ request to validate the data stored in a DX

Storage cluster (

any

cluster). By supplying the URL returned in the Location header from the

WRITE request (perhaps replacing the host address if connected to a different cluster or node), the

application can ask DX Storage to validate while reading the data. For example:

GET http://129.69.251.143/41A140B5271DC8D22FF8D027176A0821?hashtype=md5&

hash=7A25E6067904EAC8002498CF1AE33023 HTTP/1.1

Upon receiving such a READ request, DX Storage recomputes the hash of the stored content using

the supplied hashtype and compares the computed hash with the hash value in the integrity seal. If

the two values do not match, DX Storage send a 200 (OK) response but drops the connection prior

to sending the object content. If, on the other hand, the content has not been modified or corrupted

in any way, the hashes match and, in that case, DX Storage returns the object as usual with the

computed digest as a trailing Location header.

Because the hash algorithms are published and well-known, it is also possible for users and third

parties to independently validate an object that has been stored by DX Storage. This can be done

by reading its contents, computing the hash value, and comparing it with the hash value in the seal.

By publishing an integrity seal at the time it is created one can even provide indisputable evidence