Remote Server Sync

---

The Files.com Remote Server Sync feature gives you the ability to push or pull files to or from remote servers.

The remote server types that we currently support include: Azure Blob Storage, Backblaze B2, Box, Dropbox, FTP, Google Cloud Storage, Google Drive, OneDrive, Rackspace Cloud Files, Amazon S3, SFTP, SharePoint, Wasabi, and WebDAV.

The Remote Server can be a third party cloud, another Files.com account, an on-premise server running the Files.com On-Premise Agent, or any server accessible via FTP, SFTP, WebDAV, S3 protocol or anything else. See the rest of this documentation for a list of all the accessible remote server types.

A remote sync can be a "push", where files from your Files.com site are transferred to the remote server, a "pull", where files are transferred from the remote server to your Files.com site, or a two-way "sync", where files that are new or changed in either location are pushed and pulled to each other to maintain a synchronized state between the folder on your Files.com site and the folder on the remote server.

Any Remote Server sync will terminate if it runs for 24 hours. In most cases a new sync will then start to pick up where it left off when configured as a timed trigger.

Files.com's Sync functionality runs in Jobs, which are scheduled on regular intervals, with a minimum interval length determined by your subscription plan. Syncs may be run every 15 minutes for customers on our Power plan, and every 1 to 5 minutes for customers on our Premier plan. You may also increase, but not decrease, these times. Syncs are not included in our Legacy plans.

You will find the option to increase the time between successive runs for a sync in the settings for that sync. You can increase the duration by adjusting the value in the "Synchronization interval (in minutes)" field.

If you are concerned about precise timing, we recommend our Remote Server Mount feature instead, which provides a realtime view into the Remote Server via a Files.com folder.

With a Mount, files are only stored on the remote server and not in Files.com. All file operations such as upload, download, list, and so on, are passed through to the remote server in real time. You will find the Mount option in Folder settings, and site administrators can view all mounted folders by typing "Remote server mount" in the search box at the top of every page and then clicking on the matching result.

When a Sync job runs, Files.com will list the relevant folders on both the source and destination of the sync and look for files that are missing or where the size is reported to be different on the source and destination. In these cases, Files.com will copy the file from the source to the destination. For "push" and "pull" configurations, Files.com can optionally delete the file from the source after a successful copy.

Files may be transferred in any order, and parallelism is used where possible. To limit the maximum parallelism used by Files.com, you may set a setting on the Remote Server to define the maximum number of parallel connections.

Files.com only uses filename and size to determine whether a file needs to be synced. It does not currently use information such as modified date or checksum for this purpose.

Due to the wide variety of remote server types supported (and our plans to support hundreds more via an open integration platform coming soon), this is the only methodology we've been able to design that works the most consistently across all remote server types.

This strategy seems to work for 98% of the syncs we actually encounter at Files.com. There are a few cases where the use of Remote Server Sync does not work well.

Syncing Microsoft Excel Files to Microsoft SharePoint - In the case of Microsoft, they state that when uploading files to SharePoint Document Library_,_ SharePoint stores the associated column and column value of the file in the file. So you will find that the file size has changed.

3play's FTP server - As for 3play's FTP server, it has been found to report sizes incorrectly.

We are working on additional functionality to allow sync to still work intelligently in these use cases.

In the meantime, we recommend that you disable Sync and instead use a Remote Server Mount in conjunction with a Copy File or Move File Automation in either of these cases.

We would be interested to learn about other customer use cases where this strategy is not adequate. Please contact us with more information about your specific needs.

Files will run one sync job at a time per configured sync. Sync jobs run until they are complete. The larger the folder being synced, the longer sync jobs will take to complete. In practice, the majority of sync jobs complete very quickly (on the order of seconds), but we have some customers whose sync jobs take hours.

A sync job runs one control process, which runs in a single thread (i.e. not parallelized) to determine which files need to be transferred. The actual file transfers are parallelized to achieve faster sync throughput.

Said another way, you can expect that we will send multiple file upload and download requests at once, but we will not send multiple folder list requests at once.

The overall time that a sync job takes depends on how many files and folders are being inspected, the quantity of files requiring transfer, the size of the files, and the transfer speed of the remote server. Consumer-oriented remote servers, such as Dropbox and Google Drive, implement stricter rate limiting which results in slower syncs. Enterprise-oriented remote servers such as Azure, AWS, and GCP tend to have better throughput capabilities and therefore complete much more quickly.

Remote servers that run FTP or SFTP have performance that is widely variable depending on who is operating the remote and what software it runs. Where possible, we recommend using FTPS instead of SFTP, as it is usually many times faster than SFTP.

There are 2 limits related to folder size as it relates to Sync. There is a soft limit of 10,000 files per folder, and a hard limit of 100,000 files per folder.

One easy way to stay under the 10,000 item per folder limit is to divide up a folder's contents into subfolders. You can choose any division scheme that makes sense for your application, such as dividing by date, by customer, or even by filename prefix.

Our Automations feature can be used to move files into subfolders on an hourly or daily basis.

Another method that may help to manage a folder's file count is the File Expiration folder setting. This feature will automatically delete files when they reach a specified age, so that older files that are no longer needed are automatically removed.

If you exceed the soft limit of 10,000 files per folder, the sync will perform slowly and may be deprioritized behind other syncs. If you exceed the hard limit of 100,000 files per folder, the sync will fail entirely.

Due to the wide variety of remote server types supported, Files.com does not check data integrity when files are transferred during sync.

We rely on the next run of the sync to again check whether the file size and file name match on the source and destination of the sync. If a file is found to be the wrong size at that time, Files.com will attempt to resync the file.

We intend to improve upon this in future improvements to the Files.com service, where possible on the remote.

We have heard from one customer case where the remote server does not correctly report the size of files. Unfortunately, Files.com requires the remote server to accurately report the size of all files in order for Syncs to work correctly. Otherwise, the remote server will not be compatible for use with the Remote Sync feature.

Files.com records a complete log of every sync, including details about which folders were listed, which files were transferred, and which files were skipped.

You can browse these logs using the web UI. Type "External logs" in the search box at the top of every page, and then click on the matching result.

Additionally, you can download the raw logs for processing on your side. They are formatted as text files where each line of the file is a JSON hash containing the information about a particular log item. You should process the file line by line, and then run each line through a JSON parser.

We also provide a rich API for querying these logs. This API offers many filtering options.

We are working on the ability to export these logs to external SIEM services for our Enterprise customers. if this is something you would be interested in, please let us know.

When errors are detected in a sync, Files.com will automatically alert site admins via email. You can also use the External Events API to query for failed external events.

Failures are retried automatically by Files.com. If a Sync fails, it will be logged as an error once all retry attempts have been exhausted. Each scheduled Sync is independent, so one scheduled Sync failing will not prevent the next scheduled Sync from attempting to run.

Sync requires that the user or service account that you are using on the remote server has at least read, write, and list permissions on the targeted folder and its sub-folders. If a sync is only performed in one direction, read or write permission may be omitted in the direction that isn't being used.

Sync depends on being able to list the remote server in order to know which files need to be synced.

If you do not have list permissions on the remote server, we recommend that you use our Remote Server Mount feature.

To setup a Remote Server Mount to replace a remote sync, create a new folder in the same location as the synced folder and navigate to the new folder. Click the Folder settings button on the top right, and select Remote server mount. Click the Add new remote server mount button. Select the remote server that you are using for the sync, and choose the remote folder. After saving the new remote mount, remove the synced folder.

If necessary, you can then set up an Automation to trigger that files be moved into the mount based on whatever triggers may be required.

In order to sync files or folders, they must comply with any filename restrictions on the destination folder, as well as overall requirements for filenames on Files.com. For example, Files.com disallows spaces at the end of file/folder names.

If Files.com encounters files that would not be allowed on the Files.com side, the sync will result in a Partial Failure.

Additionally, Files.com is a case-insensitive file system, meaning that you should be very cautious when using a Sync in conjunction with a case-sensitive file system, such as certain Linux-based servers.

Files.com will use a best-effort to have case conversions work intelligently, however, it's impossible to perfectly model an ideal behavior in this situation.

Adding remote servers in Files.com is easy. As a Site Administrator, type "Remote servers" in the search box at the top of every page, and then click on the matching result. Click on Add new remote server to select the type of server you want to add. Only Site Administrators can add or configure remote servers.

Depending on the type of server you are adding, you may need to have different details on hand.

When you select the remote server type, the form will adapt to prompt you for the information needed specifically for that type. Any optional settings for your selected type will also appear on the form.

Once you have one or more remote servers defined, you are ready to add the remote server syncs to initiate the operation.

Type "Sync to/from remote server" in the search box at the top of every page and then click on the matching result.

Click the Add new remote server sync button to reveal the form.

Select the server you would like to transfer to or from by clicking on the Remote server menu.

For Sync, there are several options available to indicate the direction of the Sync. You have three choices:

1. Push to the remote server: This option uploads files and folders from the specified folder in your Files.com site and saves them to the remote server.
2. Pull from the remote server: This option downloads files from the remote server and saves them in the specified folder in your Files.com site.
3. Two-way sync: This option will push files from either server to the other so that both servers have the complete collection of all files. If the same file name exists on both servers with different file contents (as determined by file size), the last modification date is used to determine which version to keep. The file with the most recent modification date is then pushed to the other server. This option does not ever delete files from either server.

You also have the option to delete files on the source server after a push or pull. Use the After copying menu to select whether you would like files that are successfully transferred to be deleted from or kept on the source server. If you choose the delete option then only files in the source folder will be deleted while the corresponding files in the destination folder will remain unaffected.

If you are syncing files into a folder on Files.com that has folder settings enabled that transform the contents or filenames of files (such as GPG Encryption, GPG Decryption, or Filename rewriting rules), it is required that you enable the option to delete files from the source as part of the sync. Otherwise, the file would sync every single time the sync process runs, causing you a high transfer usage bill.

When delete source files is selected, you have an option to delete empty folders from source once done syncing as an option to clean up empty resulting folder.

Enter the remote path to or from which you would like files and folders transferred, starting after the folder/directory your remote user lands in upon authentication.

For example: if the remote server has a folder structure "folderA/folderB/folderC_"_, and the user credentials that you have configured your sync server to log in with automatically land that user inside "folderA", then to properly configure your sync folder behavior to transfer files to or from "folderC", you would enter the path as "folderB/folderC".

If you are adding this sync behavior from the Sync to/from remote server page, then you will also need to specify which folder in your Files.com site you would like to apply the sync on using the Limit to a specific folder link.

There are no limits for the number of Syncs that you can configure on the Files.com platform.

However, the remote servers that you are connecting with are very likely to limit the number of connections made to them. Contact the system administrator of the remote servers to determine what their limits are.

Maintaining a one-to-one relationship between Syncs and remote servers will allow you to have as many Syncs as you require. For example, if you have 50 remote servers then you can have 50 Syncs, each to one of the remote servers, without any issues.

When there is a many-to-one relationship between Syncs and a remote server, then the number of Syncs will be limited by the limitations and performance of the remote server. For example, if you have 50 Syncs set up with the remote server, and it only supports 5 concurrent connections, then only 5 of the Syncs will work at a time.

Finding the exact number of Syncs for optimal performance is more of an art than a science, due to factors such as automatic retries, sync intervals, folder depth and breadth, quantity of files, size of files, network performance, and remote system performance. Calibrate your Sync settings, such as time intervals between syncs, until you find the optimal settings that work best with the remote server.

Files.com enforces two limits on customers who have lots of syncs configured on the same account. No more than 5 sync jobs can start per minute, and no more than 5 sync jobs can run at a time.

As an example, we have one customer with 1,800 sync jobs configured. Regardless of the sync interval they've specified, each sync job will run no more than once every 6 hours, because 1,500 (# of syncs) / 60 (minutes per hour) / 5 (max sync jobs per minute) = 6 hours.

Like all limits on Files.com, we are happy to provide increased limits if your needs require. Please contact us for further information.

Access the list of remote syncs by typing "Sync to/from remote server" in the search box at the top of the page, and then clicking on the matching result. To enable or disable a remote sync, click the Edit button for the remote sync that you want to affect, click the toggle at the top-right of the configuration section, then click Save.

Disabled items in the Sync to/from remote server section will display a "pause" icon next to their name.

Deleting a folder that has been configured for Remote Sync will also delete its associated Remote Sync. If the folder is deleted, the remote sync will no longer exist nor function.

If the folder is deleted, the entries for actions performed by the Remote Sync will remain in the History and External logs.

When a remote server sync operation fails, site admins who have not disabled site alert emails will receive an email notification. The site alert emails may also include other types of important notifications, such as key expirations or problems with SSO, but all sync related items will be included in a list titled REMOTE SERVER SYNC FAILURES. If there are multiple failed remote syncs, all of them will be included in the same email. The email includes a link to access the details of each failed sync in the external events log along with the time of the failure.