Remote Server Sync


The Files.com Remote Server Sync feature gives you the ability to push or pull files to or from remote servers.

The remote server types that we currently support include: Azure Blob Storage, Backblaze B2, Box, Dropbox, FTP, Google Cloud Storage, Google Drive, OneDrive, Rackspace Cloud Files, Amazon S3, SFTP, SharePoint, Wasabi, and WebDAV.

A remote sync can be a "push", where files from your Files.com site are transferred to the remote server, a "pull", where files are transferred from the remote server to your Files.com site, or a two-way "sync", where files that are new or changed in either location are pushed and pulled to each other to maintain a synchronized state between the folder on your Files.com site and the folder on the remote server.

How Syncs Work on Files.com

Files.com's Sync functionality runs in Jobs, which are scheduled on regular intervals, with a minimum interval length determined by your subscription plan. Syncs may be run every hour for customers on our Starter plan, every 15 minutes for customers on our Power plan, and every 5 minutes for customers on our Premier plan. You may also increase, but not decrease, these times.

You will find the option to increase a syncs interval length at Settings > Integrations > Sync to/from remote server. You can increase the duration by adjusting the value in the "Synchronization interval (in minutes)" field.

If you are concerned about precise timing, we recommend our Remote Server Mount feature instead, which provides a realtime view into the Remote Server via a Files.com folder.

With a Mount, files are only stored on the remote server and not in Files.com. All file operations such as upload, download, list, and so on, are passed through to the remote server in real time. You will find the Mount option in Folder settings, and at Settings > Integrations > Remote server mount.

When a Sync job runs, Files.com will list the relevant folders on both the source and destination of the sync and look for files that are missing or where the size is reported to be different on the source and destination. In these cases, Files.com will copy the file from the source to the destination. For "push" and "pull" configurations, Files.com can optionally delete the file from the source after a successful copy.

Files may be transferred in any order, and parallelism is used where possible. To limit the maximum parallelism used by Files.com, you may set a setting on the Remote Server to define the maximum number of parallel connections.

Determining When To Sync (Transfer) A File

Files.com only uses filename and size to determine whether a file needs to be synced. It does not currently use information such as modified date or checksum for this purpose.

Due to the wide variety of remote server types supported (and our plans to support hundreds more via an open integration platform coming soon), this is the only methodology we've been able to design that works the most consistently across all remote server types.

Known Limitations

This strategy seems to work for 98% of the syncs we actually encounter at Files.com. There are a few cases where this strategy does not work, and you should not use Files.com's Remote Server Sync feature. These include:

  • Syncing Microsoft Excel Files to Microsoft SharePoint. Microsoft states that when uploading files to SharePoint Document Library, SharePoint stores the associated column and column value of the file in the file. So you will find that the file size has changed.
  • 3play's FTP server has been found to report sizes incorrectly.

We are working on additional functionality to allow sync to still work intelligently in these use cases.

In the meantime, we recommend that you disable Sync and instead use a Remote Server Mount in conjunction with a Copy File or Move File Automation in either of these cases.

We would be interested to learn about other customer use cases where this strategy is not adequate. Please contact us with more information about your specific needs.

Parallelism, Scheduling and Sync Speeds

Files will run one sync job at a time per configured sync. Sync jobs run until they are complete. The larger the folder being synced, the longer sync jobs will take to complete. In practice, the majority of sync jobs complete very quickly (on the order of seconds), but we have some customers whose sync jobs take hours.

A sync job runs one control process, which runs in a single thread (i.e. not parallelized) to determine which files need to be transferred. The actual file transfers are parallelized to achieve faster sync throughput.

Said another way, you can expect that we will send multiple file upload and download requests at once, but we will not send multiple folder list requests at once.

The overall time that a sync job takes depends on how many files and folders are being inspected, the quantity of files requiring transfer, the size of the files, and the transfer speed of the remote server. Consumer-oriented remote servers, such as Dropbox and Google Drive, implement stricter rate limiting which results in slower syncs. Enterprise-oriented remote servers such as Azure, AWS, and GCP tend to have better throughput capabilities and therefore complete much more quickly.

Remote servers that run FTP or SFTP have performance that is widely variable depending on who is operating the remote and what software it runs. Where possible, we recommend using FTPS instead of SFTP, as it is usually many times faster than SFTP.

Folder Size Limitations

There are 2 limits related to folder size as it relates to Sync. There is a soft limit of 10,000 files per folder, and a hard limit of 100,000 files per folder.

One easy way to stay under the 10,000 item per folder limit is to divide up a folder's contents into subfolders. You can choose any division scheme that makes sense for your application, such as dividing by date, by customer, or even by filename prefix.

Our Automations feature can be used to move files into subfolders on an hourly or daily basis.

Another method that may help to manage a folder's file count is the File Expiration folder setting. This feature will automatically delete files when they reach a specified age, so that older files that are no longer needed are automatically removed.

If you exceed the soft limit of 10,000 files per folder, the sync will perform slowly and may be deprioritized behind other syncs. If you exceed the hard limit of 100,000 files per folder, the sync will fail entirely.

Data Integrity

Due to the wide variety of remote server types supported, Files.com does not check data integrity when files are transferred during sync.

We rely on the next run of the sync to again check whether the file size and file name match on the source and destination of the sync. If a file is found to be the wrong size at that time, Files.com will attempt to resync the file.

We intend to improve upon this in future improvements to the Files.com service, where possible on the remote.

We have heard from one customer case where the remote server does not correctly report the size of files. Unfortunately, Files.com requires the remote server to accurately report the size of all files in order for Syncs to work correctly. Otherwise, the remote server will not be compatible for use with the Remote Sync feature.

Logging and Error Handling

Files.com records a complete log of every sync, including details about which folders were listed, which files were transferred, and which files were skipped.

You can browse these logs using the web UI by navigating to Settings > Logs > External.

Additionally, you can download the raw logs for processing on your side. They are formatted as text files where each line of the file is a JSON hash containing the information about a particular log item. You should process the file line by line, and then run each line through a JSON parser.

We also provide a rich API for querying these logs. This API offers many filtering options.

We are working on the ability to export these logs to external SIEM services for our Enterprise customers. if this is something you would be interested in, please let us know.

When errors are detected in a sync, Files.com will automatically alert site admins via E-Mail. You can also use the External Events API to query for failed external events.

Failures are retried automatically by Files.com. If a Sync fails, it will be logged as an error once all retry attempts have been exhausted. Each scheduled Sync is independent, so one scheduled Sync failing will not prevent the next scheduled Sync from attempting to run.

Permissions Required on the Remote Server

Sync requires that the user or service account that you are using on the remote server has at least read, write, and list permissions on the targeted folder and its sub-folders. If a sync is only performed in one direction, read or write permission may be omitted in the direction that isn't being used.

Sync depends on being able to list the remote server in order to know which files need to be synced.

If you do not have list permissions on the remote server, we recommend that you use our Remote Server Mount feature.

To setup a Remote Server Mount to replace a remote sync, create a new folder in the same location as the synced folder and navigate to the new folder. Click the Folder settings button on the top right, and select Remote server mount. Click the Add new remote server mount button. Select the remote server that you are using for the sync, and choose the remote folder. After saving the new remote mount, remove the synced folder.

If necessary, you can then set up an Automation to trigger that files be moved into the mount based on whatever triggers may be required.

Filename Requirements

In order to sync files or folders, they must comply with any filename restrictions on the destination folder, as well as overall requirements for filenames on Files.com. For example, Files.com disallows spaces at the end of file/folder names.

If Files.com encounters files that would not be allowed on the Files.com side, the sync will result in a Partial Failure.

Additionally, Files.com is a case-insensitive file system, meaning that you should be very cautious when using a Sync in conjunction with a case-sensitive file system, such as certain Linux-based servers.

Files.com will use a best-effort to have case conversions work intelligently, however, it's impossible to perfectly model an ideal behavior in this situation.

Adding remote servers

Adding remote servers in Files.com is easy. As a Site Administrator, simply head to Integrations > Sync/Mount to select the type of server you want to add. Note that only Site Administrators can add or configure remote servers at this time.

Depending on the type of server you are adding, you may need to have different details on hand.

When you select the remote server type, the form will adapt to prompt you for the information needed specifically for that type. Any optional settings for your selected type will also appear on the form.

Adding a Sync

Once you have one or more remote servers defined, you are ready to add the remote server syncs to initiate the operation. You can do this from two locations in the web interface:

  1. From Files, navigate into the folder where you would like to add the remote server sync and click Folder settings > Sync to/from remote server.
  2. Navigate to Settings > Integrations > Sync to/from remote server (here you will need to specify the folder to which you would like to apply the remote server sync).

Click the Add new remote server sync button to reveal the form.

Select the server you would like to transfer to or from by clicking on the Remote server menu.

Sync direction

Next choose your Sync direction. You have three choices:

  1. Push to the remote server: This option uploads files and folders from the specified folder in your Files.com site and saves them to the remote server.
  2. Pull from the remote server: This option downloads files from the remote server and saves them in the specified folder in your Files.com site.
  3. Two-way sync: This option will push files from either server to the other so that both servers have the complete collection of all files. If the same file name exists on both servers with different file contents (as determined by file size), the last modification date is used to determine which version to keep. The file with the most recent modification date is then pushed to the other server. This option does not ever delete files from either server.

Delete or Keep after copying

You also have the option to delete files on the source server after a push or pull. Use the After copying menu to select whether you would like files that are successfully transferred to be deleted from or kept on the source server. If you choose the delete option then only files in the source folder will be deleted while the corresponding files in the destination folder will remain unaffected.

Note that only files will be deleted, not folders. There is a feature request filed to also support deleting empty folders during a sync process. We will update this page when that feature is available. If this is a feature that would be helpful for you, please contact us and let us know so that we may add your name to the list.

If you are syncing files into a folder on Files.com that has folder settings enabled that transform the contents or filenames of files (such as GPG Encryption, GPG Decryption, or Filename rewriting rules), it is required that you enable the option to delete files from the source as part of the sync. Otherwise, the file would sync every single time the sync process runs, causing you a high transfer usage bill.

Remote path

Enter the remote path to or from which you would like files and folders transferred, starting after the folder/directory your remote user lands in upon authentication.

For example: if the remote server has a folder structure folderA/folderB/folderC, and the user credentials that you have configured your sync server to log in with automatically land that user inside folderA, then to properly configure your sync folder behavior to transfer files to or from folderC, you would enter the path as folderB/folderC.

NOTE: If you are adding this sync behavior from the settings page at Settings > Integrations > Sync to/from remote server, then you will also need to specify which folder in your Files.com site you would like to apply the sync on using the Limit to a specific folder link.

Number of Syncs

There are no limits for the number of Syncs that you can configure on the Files.com platform.

However, the remote servers that you are connecting with are very likely to limit the number of connections made to them. Contact the system administrator of the remote servers to determine what their limits are.

Maintaining a one-to-one relationship between Syncs and remote servers will allow you to have as many Syncs as you require. For example, if you have 50 remote servers then you can have 50 Syncs, each to one of the remote servers, without any issues.

When there is a many-to-one relationship between Syncs and a remote server, then the number of Syncs will be limited by the limitations and performance of the remote server. For example, if you have 50 Syncs set up with the remote server, and it only supports 5 concurrent connections, then only 5 of the Syncs will work at a time.

Finding the exact number of Syncs for optimal performance is more of an art than a science, due to factors such as automatic retries, sync intervals, folder depth and breadth, quantity of files, size of files, network performance, and remote system performance. Calibrate your Sync settings, such as time intervals between syncs, until you find the optimal settings that work best with the remote server.

Limitations Associated With High Numbers of Syncs

Files enforces two limits on customers who have lots of syncs configured on the same account. No more than 5 sync jobs can start per minute, and no more than 5 sync jobs can run at a time.

As an example, we have one customer with 1800 sync jobs configured. Regardless of the sync interval they've specified, each sync job will run no more than once every 6 hours, because 1500 (# of syncs) / 60 (minutes per hour) / 5 (max sync jobs per minute) = 6 hours.

Like all limits on Files.com, we are happy to provide increased limits if your needs require. Please contact us for further information.

Enabling and disabling a remote sync

To enable or disable a remote sync, navigate to Settings > Integrations, scroll down to the Sync to/from remote server section, click the Edit button for the remote sync that you want to affect, click the toggle at the top-right of the configuration section, then click Save.

Disabled items in the Sync to/from remote server section will display a "pause" icon next to their name.

Deleting a Remote Sync Folder

Deleting a folder that has been configured for Remote Sync will also delete its associated Remote Sync. If the folder is deleted, the remote sync will no longer exist nor function.

If the folder is deleted, the entries for actions performed by the Remote Sync will remain in the History and External logs.

Site Alert Emails

When a remote server sync operation fails, site admins who have not disabled site alert emails will receive an email notification. The site alert emails may also include other types of important notifications, such as key expirations or problems with SSO, but all sync related items will be included in a list titled REMOTE SERVER SYNC FAILURES. If there are multiple failed remote syncs, all of them will be included in the same email. The email includes a link to access the details of each failed sync in the external events log along with the time of the failure.

Troubleshooting

There a several issues that can cause remote sync failures, such as:

  • authentication issues
  • firewall issues on the remote server
  • general remote server issues
  • invalid file names
  • invalid file sizes
  • slow transfer times

When these issues occur, you may see effects such as some files successfully being transferred while others fail. We provide detailed error messages that should allow you to determine the cause of the failure.

Authentication or Firewall issues

Authentication issues can usually be resolved by correcting any relevant credentials. However, sometimes authentication issues can occur because of IP address requirements.

Be aware of the following if the remote server enforces an IP address whitelist:

  • If your site uses a custom domain, you have two dedicated IPs that may be used for outbound connections. You can find your dedicated IPs by going to Settings > Integrations and scroll to Firewall configuration. However, you can disable the use of your dedicated IP in these circumstances if you need to. (You might do that if your counterparty has already whitelisted the main Files.com IP range, for example.)
  • If you do not have a custom domain, ensure that our main IPs on this list are whitelisted, not just some of them. There are quite a lot of IPs on that list (over 80 at last count) and you need to whitelist all IPs or else you will experience failures. If whitelisting that many IP addresses is a problem for you, the solution is to move to a custom domain. This will get you a pair of IP addresses you can whitelist (see the prior bullet.)

Invalid file names

Remote servers will likely prohibit certain characters from being used in file names. This is usually caused by limitations in the underlying operating system being used. Operating systems such as Windows, Linux, and Mac each have their own specific restrictions for which characters can be used to name a file. Additionally, 3rd party service providers can overlay their own restrictions. It is not practical for Files.com to maintain a database of any restrictions that may apply on the remote.

If you see issues with invalid filenames, we recommend you simplify your filenames generally. Examples of restricted characters are:

  • < (less than)
  • \> (greater than)
  • : (colon)
  • " (double quote)
  • / (forward slash)
  • \ (backslash)
  • | (vertical bar or pipe)
  • ? (question mark)
  • \* (asterisk)

Remote servers may also restrict the length of the file name. Common length limits include:

  • 160 characters
  • 255 characters

Bear in mind that some remote servers calculate the total length as being either:

  • just the file or folder name, including any extension
  • the sum of the whole folder path + file name + extension + temporary suffix

To resolve this issue, shorten the name of files and folders so that they are below the length limits of the remote server.

On the Files.com side, the path limit is 550 characters for the sum of the full path, including folder names.

Invalid file sizes

Remote servers may have maximum size limits for files. Please check the documentation of the remote server for details on any file size limits or restrictions.

Remote servers that use APIs or HTTP based connections may also not support zero byte file sizes (for example, you can not pull zero byte size files from Azure storage).

Slow transfer times

Slow transfers and long transfer durations can be caused by low network bandwidth, high network latency, large quantity of files being transferred, huge file sizes, network throttling, remote system performance, local system performance, encryption methods, and session time limits.

The physical limitations of a network or system cannot be circumvented. For example, if sending a 10GB file across the internet to the remote server takes you 2 hours then sending the same 10GB file across the internet to the remote server will take us 2 hours also, assuming we both have identical network connectivity with the remote server.

Remote server issues

If the remote server has a temporary issue, such as being temporarily offline or unavailable, then the current sync will fail but subsequent scheduled syncs will retry until all syncs have been completed.

If the remote server has a permanent issue or limitation, such as not accepting file sizes greater than 2GB or having the incorrect access permissions set, then affected files will never be successfully transferred and subsequent runs of the sync will keep repeating the transfer attempt and will keep showing failure outcomes in the logs.

To resolve these issues, make sure that the correct access permissions have been set on the remote server and that any known limitations are communicated to your users. For example, if the remote server cannot accept file sizes greater than 2GB then communicate to your users that file sizes should be kept below the 2GB limit.

To troubleshoot access permission issues, configure a connection to the remote server using credentials with full (read, write, update, delete, list, etc.) access permissions and see if the issue persists. If not, then an access permission is causing the issue. Reconfigure the access permissions one at a time and re-test the sync to determine exactly which access permission is causing the issue.

Connection timeout

If the connection to the remote server times out then the current sync run will finish and report a Failure or Partial Failure error.

Subsequent syncs will attempt to continue transferring the remaining files, omitting those files that have already completed successfully.

Much like the slow transfer times issues, if the transfer duration is longer than the connection timeout, then the next run could restart the transfer over from the beginning, causing an endless retry loop where the file never gets completely transferred to the remote server.

To resolve this issue, contact the administrator of the remote server and ask if they can configure a longer connection timeout duration or limit the file sizes of your files so that they can be fully transferred within the duration of the connection.

Get Instant Access to Files.com

The button below will take you to our Free Trial signup page. Click on the white "Start My Free Trial" button, then fill out the short form on the next page. Your account will be activated instantly. You can dive in and start yourself or let us help. The choice is yours.

Start My Free Trial

©2022 Files.com. All right reserved

FILES.COM

  • Start My Free Trial
  • Pricing
  • Docs
  • API and SDKs
  • Contact

CONTACT & SUPPORT

support@files.com

(800) 286-8372

Monday–Friday

9am–8pm Eastern