Amazon S3

Both Standard and Standard - IA storage is supported. Refer to AWS documentation for the differences between these storage types.

Note: All data in transit to and from a storage node, and stored in the storage node is encrypted. Do not use the AWS encryption on the S3 bucket.

Bucket Name – The unique name of the Amazon S3 bucket that you want to add as a storage node.
Use Access and Secret Keys – Use Amazon S3 access credentials for the storage node.

• Access Key ID – The Amazon S3 access key ID.
• Secret Access Key – The Amazon S3 secret access key.

Use AWS IAM Role Policy – When the portal is also running as an AWS EC2 instance, you can define an IAM policy and then assign this policy to an EC2 role which is then attached to the portal instance, via Instance Settings > Attach/Replace IAM Role in the AWS Management Console. If you set up this type of policy, you do not need to specify the Access and Secret keys to access the storage node.
Endpoint – The endpoint name of the S3 service. The default value for Amazon S3 is s3.amazonaws.com. The port for the endpoint can be customized by adding the port after the URL, using a colon (:) separator. The default port is 80.
Use HTTPS – Use HTTPS to connect with the storage node.

Trust all certificates – Trust any security certificate presented by the storage node. 

Direct Mode – Data is uploaded and downloaded directly to and from the storage node and not via the portal. If direct mode is defined for the storage node, CTERA recommends setting the deduplication method to fixed blocks and keeping the default 4MB fixed block size. For details, see Default Settings for New Folder Groups.

Add Metadata Tags – Use custom metadata to support information lifecycle management rules (ILM) on the storage node. Checking Add Metadata Tags implements the ILM, enabling storage tiering so that data can be routed across different object storages.

Integrating CTERA Portal with S3-Versioned Buckets

This document describes the procedure to use with AWS S3 storage buckets when versioning is used for CTERA Portal data. When versioning is enabled, you want to ensure that versioned objects are retained for seven (7) days. To ensure that versioned data is deleted from the S3 bucket at the same time that the metadata is deleted from the portal database, you need to create a lifecycle rule in AWS.

Note: If, for any reason, you want to keep versioning for longer than seven days, you can choose a number greater than seven

To set up versioning:

1. From your Amazon Web Services account, sign-in to the AWS Management Console and select Services.
2. Under Storage, select S3.
3. Click the CTERA Portal bucket from the S3 buckets list and then select the properties tab to check that Versioning is enabled.
   The properties for the bucket are displayed.
   
   When Versioning is enabled, you are paying for multiple versions of the same document, which you will want to remove in line with the CTERA Portal retention policy.
4. Select the Management tab.
   The management details for the bucket are displayed.
   
5. Click Add lifecycle rule.
   The Lifecycle rule wizard is displayed.
   
6. Enter a name for the rule and select the Apply to all objects in the bucket option.
7. Click Next.
   The Transitions step is displayed.
   
8. Select Previous versions.
   After checking Previous versions, the Transitions step is redisplayed with the option to add a transition storage type.
   
9. Without adding a transition, click Next.
   The Expiration step is displayed.
   
10. Select Previous versions.
    The Expiration step is redisplayed with the option to specify the number of days to wait before deleting a previous version.
    
11. Specify seven (7) days, after which the previous version is permanently deleted.
12. Check Clean up expired object delete markers.
13. Click Next.
    The Review step is displayed.
    
14. Review the details of the rule and if satisfied, check the acknowledgment.
15. Click Save.

Lifecycle rules run once a day at midnight UTC.

Using AWS Intelligent Tiering For CTERA Portal Storage

Each object in Amazon S3 has a storage class associated with it. Amazon S3 offers a range of storage classes for the objects that you store. You choose a class depending on your use case scenario and performance access requirements.

The STANDARD storage class is the default storage class if you don't specify a storage class when you upload an object to AWS. However, Amazon also offer a storage class that automatically optimizes frequently and infrequently accessed objects, the INTELLIGENT_TIERING storage class.

The INTELLIGENT_TIERING storage class is designed to optimize storage costs by automatically moving data to the most cost-effective storage access tier, without performance impact or operational overhead.

How Does Intelligent Tiering Work?

The INTELLIGENT_TIERING storage class is suitable for objects larger than 128 KB that you plan to store for at least 30 days. The storage class stores objects in two access tiers: one tier that is optimized for frequent access and another lower-cost tier that is optimized for infrequently accessed data. Amazon S3 monitors access patterns of the objects in the storage class and moves data on a granular object level that has not been accessed for 30 consecutive days to the infrequent access tier.

With intelligent tiering, you are charged a monthly monitoring and automation fee per object instead of retrieval fees. If an object in the infrequent access tier is accessed, it is automatically moved back to the frequent access tier, but no fees are applied when objects are moved between access tiers within the INTELLIGENT_TIERING storage class.

The bigger the block size, the larger the savings. CTERA recommends using the INTELLIGENT_TIERING storage class when the block size is set to 1MB or larger. If the block size is less than 1MB, contact CTERA support to see whether there is a saving. The larger the average object size the more negligible is the monitoring and automation fee as part of the whole fee. Whether you use intelligent tiering or not is mainly dependent on the following considerations:

• The average block size of your objects. The INTELLIGENT_TIERING storage class is suitable for objects larger than 128KB. In CTERA Portal files are broken down in to blocks and the block size is controlled by the Average Block Size setting in the Virtual Portal Settings. The default average block size is 512KB.
  Note

  Use Fixed Block Size if direct mode is defined for the storage node and CTERA recommends keeping the default 4MB fixed block size.

  

  Note

  CTERA Portal file maps are typically small and are not included in the intelligent tiering transition rule.
• The percentage of infrequently accessed data.
• The percentage of objects stored for less than 30 days. The INTELLIGENT_TIERING storage class is suitable for objects that you plan to store for at least 30 days and if you delete an object before the end of the 30-day minimum storage duration period, you are charged for 30 days. CTERA Portal retains deleted files for at least 30 days, to enable undeleting a file, meaning that this requirement can be ignored.

Changing Storage to Intelligent Tiering

Before transitioning storage to the INTELLIGENT_TIERING storage class, CTERA recommends checking the average size of the objects being stored and the days they are held in storage.

To transition storage to intelligent tiering storage:

1. From your Amazon Web Services account, sign in to the AWS Management Console and select Services.
2. Under Storage , select S3.
3. Click the CTERA Portal bucket from the S3 buckets list and then select the Management tab.
   The management details for the bucket are displayed.
   
4. Click Add lifecycle rule.
   The Lifecycle rule wizard is displayed.
   
5. Enter a name for the rule and in the filter text box enter blocks and the press Enter.
   
6. Click Next.
   The Lifecycle rule wizard Transitions screen is dis played.
   
7. Check Current version.
   The Transitions screen is changed to include the ability to add transitions.
8. Click Add transition and from the Object creation drop-down select Transition to Intelligent-Tiering after.
   
9. Specify 7 days to wait until the transition is performed.
10. Click Next.
    The Lifecycle rule wizard Expirations screen is dis played.
    
11. Leave the defaults, everything unchecked, and click Next.
    The Lifecycle rule wizard Review screen is dis played.
    
12. Click Save.
    The management details for the bucket are displayed showing the added lifecycle rule.
    
    Use management metrics to validate the storage transition.

To access management metrics for an S3 bucket:

1. From your Amazon Web Services account, sign in to the AWS Management Console and select Services.
2. Under Storage, select S3.
3. Click the CTERA Portal bucket from the S3 buckets list and then select the Management tab.
   The management details for the bucket are displayed.
   
4. Click Metrics.
   The metrics for the bucket are displayed. Graphs are displayed after transitioning to the INTELLIGENT_TIERING storage class.