Determining When To Sync A File

Files.com uses filename and file size to determine whether a file needs to be synced. It does not currently use information such as modified date or checksum for this purpose.

This methodology works consistently across the wide variety of remote server types Files.com supports.

Checking For Updated File

Syncs with object storage remote servers of type AWS S3, Azure Blob, or Google Cloud Storage include an extra file update check.

The sync checks whether a file has been updated since the sync started. If the file was updated, it is skipped from this sync. This prevents partial files from being delivered to the destination while the source is still being written.

A file skipped for this reason records the following message in the Sync Logs:

Detected a change in file during sync

Skipped files are processed again in subsequent syncs. When the same file is skipped run after run, it is being written continuously, or it is being written on the same schedule the sync runs on. Move the sync to a window when nothing is writing to the file.

Known Limitations

The sync strategy works for over 98% of the use cases Files.com has encountered. A few cases exist where Sync does not work well.

Syncing Files to Microsoft SharePoint

Microsoft SharePoint parses files in order to index and promote them within the SharePoint user interface. Files uploaded to a Document Library are modified by SharePoint to include extra column and value metadata. The file size increases and its modification time is updated, which makes Sync unable to determine if the synced file matches the original.

Open Files and 3rd Party File Locks

A file lock prevents a file from being read while it is being modified or deleted. Locks only work when a single controller manages the lock, and file locking across networked or shared systems is not consistently implemented, so some parties never learn that a lock exists.

Mainframes and Windows provide built-in file locks. Unix, Linux, and macOS do not. Applications and scripts may also implement their own locking, which other applications and systems are unaware of, though you can often detect it by the presence of lock files.

Files.com sees only its own internal locks and the locks a remote server or its connection protocol explicitly publishes, so a lock on a remote server is often invisible to Files.com. A locked file on a Windows remote server cannot be synced at all, because the operating system prevents it from being read. A file being modified on a Linux remote server syncs anyway, and the data that reaches the destination is partial or invalid.

Never use a location that holds open files as the source or destination of a Sync or an Automation.

3play's FTP Server

3play's FTP server reports file sizes incorrectly. Because Sync uses file size to determine whether a file needs to be transferred, syncs against 3play's FTP server produce unexpected outcomes.

Mitigating the Limitations

In these cases, disable Sync and instead use a Remote Server Mount in conjunction with a Copy File or Move File Automation.

If you have a use case where this strategy is not adequate, contact us with the details.