Determining When To Sync A File

Files.com uses filename and file size to determine whether a file needs to be synced. It does not currently use information such as modified date or checksum for this purpose.

This methodology works consistently across the wide variety of remote server types Files.com supports.

Checking For Updated File

Syncs with object storage remote servers of type AWS S3, Azure Blob, or Google Cloud Storage include an extra file update check.

The sync checks whether a file has been updated since the sync started. If the file was updated, it is skipped from this sync. This prevents partial files from being delivered to the destination while the source is still being written.

A file skipped for this reason records the following message in the Sync Logs:

Detected a change in file during sync

Skipped files are processed again in subsequent syncs. When the same file is skipped run after run, it is being written continuously, or it is being written on the same schedule the sync runs on. Move the sync to a window when nothing is writing to the file.

Known Limitations

Filename and size comparisons have limits when another system changes files or reports their metadata differently.

Syncing Files to Microsoft SharePoint

Microsoft SharePoint parses files in order to index and promote them within the SharePoint user interface. Files uploaded to a Document Library are modified by SharePoint to include extra column and value metadata. The file size increases and its modification time is updated, which makes Sync unable to determine if the synced file matches the original.

Open Files and 3rd Party File Locks

Files.com can only respond to locks exposed by the Remote Server or its connection protocol. A lock held by an application may not be visible through that connection. An unreadable file can fail to transfer, while a readable file that is still being written can produce an incomplete copy.

Have the producing system finish writing files before making them available in the pickup folder. Use separate working and pickup locations, or schedule transfers for a window when the files are no longer changing. A Remote Server Mount with an Automation still needs this coordination; changing the transfer mechanism does not make an unfinished file complete.

3play's FTP Server

3play's FTP server reports file sizes incorrectly. Because Sync uses file size to determine whether a file needs to be transferred, syncs against 3play's FTP server produce unexpected outcomes.

Mitigating the Limitations

In these cases, disable Sync and instead use a Remote Server Mount in conjunction with a Copy File or Move File Automation.

If you have a use case where this strategy is not adequate, contact us with the details.