- 9 Minutes to read
- Print
- PDF
Managing Caching
- 9 Minutes to read
- Print
- PDF
Content is synced between the CTERA Portal global file system and the CTERA Edge Filers connected to the CTERA Portal. Folder content is identical across all the CTERA Edge Filers connected to the CTERA Portal. Whenever any change of any kind is made to any file in any of the synced folders, the same change is made immediately in the other synced folders. For example, if a file is deleted from one of the folders, the same file is deleted from the other synced folders. It makes no difference which user made the change or in which of the synced folders the change was made.
Every 7 days a task runs in the background to verify that all the files on the edge filer are synced with the portal global file system. Any files on the edge filer that are not synced with the portal global file system are automatically synced.
Each CTERA Portal user accesses the content that was synced to the CTERA Edge Filer according to the permission set defined for their access on the CTERA Portal.
When the CTERA Portal is connected to the CTERA Edge Filer, the folder structure defined in the CTERA Portal global file system is created automatically on the CTERA Edge Filer when the CTERA Edge Filer syncs with the CTERA Portal. Setting up the CTERA Edge Filer to enable syncing is dependent on a number of considerations:
- Do you need to maintain the file structure and ACLs after migrating the data to a CTERA Edge Filer?
In this case, the setup should be done using nested shares.
Even if a folder is shared with a user but the folder has no ACL access permissions, when the user attempts to access the folder a message similar to the following is displayed:You have no permission to view this folder
. - Do you need to maintain the file structure, but not the ACLs, after migrating the data to a CTERA Edge Filer?
In this case, the setup should be done using nested shares.
Access to files is determined by the authorization levels set in the CTERA Portal, such as Read/Write or Preview Only.
CTERA recommends whenever possible to set up the CTERA Edge Filer using nested shares.
Setting up the CTERA Edge Filer for caching offers the following advantages:
- Content can be synced between different branch CTERA Edge Filers and with the CTERA Portal global file system. This means that the data is available for users to access at the branch at LAN speed, as well as when they are roaming outside the office.
- Users can sync files across their own devices. Syncing files allows users with multiple personal devices, such as a desktop computer, laptop, and smartphone, to take their files with them effortlessly wherever they go, with the confidence that the files they are working on are always current. Users do not even need to remember which device they were working on when they last modified a file. For example, you can create a file on your laptop and later open and modify it from your desktop.
- Users can collaborate easily with others. You can sync any number of shared folders and everyone who syncs the shared folder can view, update, and delete the files in the shared folder.
Storage With a Caching Gateway
With a Caching Gateway, files are stored in cloud storage and only stored on the CTERA Edge Filer when required. However, all the metadata is downloaded to the CTERA Edge Filer so users have access to everything that they have permission to access on the CTERA Portal. Changes to the metadata in the CTERA Portal are reflected in the CTERA Edge Filer, but the content itself is not stored on the CTERA Edge Filer. Only what is being worked on at the moment or what is wanted to be permanently available is stored on the CTERA Edge Filer. The files that have not been downloaded to the CTERA Edge Filer are displayed in the CTERA Edge Filer as stubs, which take up very little storage.
When a user accesses a file stub, the file is downloaded and opened without delay, and where possible, large files are streamed from the cloud so they can be accessed faster. After the download has completed, the file is unstubbed. Any changes to the file are synced back to the CTERA Portal. Folders with files that are always required can be marked in the CTERA Edge Filer user interface as pinned, in which case the files, and not the stubs, are stored on the CTERA Edge Filer. For more details about pinned files, see Pinning Folders so that Files are Always Available Locally.
Streaming is performed on all file types where the application that opens the file supports streaming.
Files on the CTERA Edge Filer are referred to as hot files. Less frequently accessed files, that remain in the cloud managed by CTERA Portal, are referred to as cold files. Files that have not been accessed recently are evicted when the amount of storage used reaches a high watermark. The CTERA Edge Filer also has spare capacity to absorb common write bursts at LAN speed. This means that users do not require access to the cloud and the additional slowness that is implied with this access is avoided during these bursts. For more details about evicting files, see File Eviction from the CTERA Edge Filer.
Any file in either the pinned data area or cache area is available at LAN speed.
Files that were updated in the CTERA Portal global file system and are defined as hot files, are downloaded automatically to the CTERA Edge Filer in background, so that when they are required, the user will have access to the up-to-date version of the file at LAN speed.
Streaming Files
Accessing a stub file from the cloud causes the file to be downloaded and opened with as little delay as possible. Where possible, files over 10MB are streamed from the cloud so that the user can access the file before it has been completely downloaded. Also, where possible, random access within the file is also supported, so the user does not have to wait for a sequential stream to complete to access parts of the file. Instead, a temp file is used and blocks are downloaded from the portal from the point the user requested.
Streamed files are added to a background download queue. After the download has completed, the file is unstubbed.
When files are retrieved directly from S3 storage (direct mode is enabled on the storage node definition in the portal), files are streamed sequentially as the files are downloaded very quickly. Random access is only implemented when the user wants to access a point in the file more than 500MB from the current point being streamed.
Files over 10MB that are saved locally are also streamed so that they can be viewed and worked on immediately.
Streaming is not dependent on the file type but on whether the application that opens the file supports streaming.
File Eviction from the CTERA Edge Filer
As the CTERA Edge Filer storage space gets used up, the CTERA Edge Filer starts to remove (evict) files leaving stubs on the CTERA Edge Filer. Files start to be evicted only when the amount of storage used reaches the high watermark, 75% of the total CTERA Edge Filer storage. At this point, files are evicted until the amount of storage used is equal to, or less than, the low water mark, 65% of the total CTERA Edge Filer storage.
The following hot files are never evicted, so they are always available at LAN speed:
- Files in a folder marked in the CTERA Edge Filer as pinned. These files are downloaded to the CTERA Edge Filer and never evicted.
- Files that are currently open.
- Files that have not yet been synced to the CTERA Portal. For example, a new file on the CTERA Edge Filer or a file that was edited but not yet synced.
Other hot files can be evicted. These files are grouped according to the time since they were last accessed. The files are evicted within these groups, the files in the oldest group first, followed by the files in the next oldest group.
Eviction is based on days and not file size. Also, there is no differentiation within a group of the length of time since a file was accessed.
For details on how to manage eviction, see Manage File Eviction From the Cache.
The following graph shows expected disk space usage in a normal scenario.
If the available free space does become too small, throttling is implemented so that users can continue to write to their files, but the client's write speed is reduced to WAN speed so that the data being written is not faster than the data that is evicted to the cloud.
This type of scenario is not usual, except during a migration of a large amount of data or a very heavy write burst, using all the 25% storage space reserved for write bursts.
What Happens to Simultaneous File Changes?
Different people can work on the same files at the same time, which means that conflicts can occur. CTERA Edge Filer keeps track of who makes updates and detects and resolves file conflicts when updates occur simultaneously. A file with the conflicts is written in the CTERA Portal global file system so that users can verify that the conflicts were resolved correctly.
CTERA Edge Filer records the history of file events. The changes made in the most recent update are always saved to the original file. Changes that are lost due to conflict are retained in a renamed version of the file.
In order for conflict resolution to be performed correctly, the CTERA Edge Filer clock must be synchronized with the CTERA Portal clock. If there is more than one hour difference between the two clocks (after taking into account time zone differences), the CTERA Edge Filer will not synchronize the cloud drive folder.
What Files are Not Synced From the CTERA Portal?
Temporary files on the CTERA Portal are not synced to the CTERA Edge Filer. The following are not synced:
- Files that begin with
~$
,.ctera.tmp
or._
- Files of type
tmp
,temp
,swp
- Files named
desktop.ini
,Thumbs.db
,.DS_Store
,._.DS_Store
,CredDB.cef
,.AppleDouble
,.AppleSingle
or.Parent
- Files that end with
Zone.Identifier
Windows ACL enabled folders on the CTERA Portal cannot be synced unless the share is defined as Windows ACL Emulation Mode, described in Configuring CTERA Edge Filer Shares.
Caching Operations
The following can be performed:
- Pinning Folders so that Files are Always Available Locally.
- Suspend and Unsuspend Cloud Sync.
- Refresh the Folder List From the CTERA Portal.
- Control Cloud Sync Upload and Download Speeds.
- Manage File Eviction From the Cache.
- Monitor Edge Filer Caching.
- Enable Fast Disaster Recovery.
- View the Cloud Sync Log.
These operations are performed from the Cloud Drive menu option in the navigation pane.