Skip to main content

Snowflake

SnowflakeExternal LinkThis link leads to an external website and will open in a new tab is a cloud-based data platform for data warehousing, analytics, and data sharing across multiple cloud providers.

Files.com integrates with Snowflake through the Files.com S3-Compatible Endpoint. Snowflake connects to your Files.com site the same way it connects to any Amazon S3 bucket, and loads data directly from Files.com folders into Snowflake. Snowflake supports this through its S3-compatible storageExternal LinkThis link leads to an external website and will open in a new tab feature, which lets you use your Files.com site's S3 endpoint as a Snowflake external stage.

Configuring Files.com

Your Files.com site includes an S3-Compatible Endpoint. Generate an S3-compatible API Key to create the access key ID and secret access key that Snowflake needs. The API Key must be associated with a user account on your site.

Grant Full folder permissions on the folder you want Snowflake to access to the user associated with the access key.

Verify that the user can access the folder and see the expected files before configuring Snowflake.

Configuring Snowflake

Refer to the Snowflake documentation on working with S3-compatible storageExternal LinkThis link leads to an external website and will open in a new tab for details on how to access your Files.com site from Snowflake.

Snowflake does not allow connections to custom S3-compatible endpoints until you contact Snowflake supportExternal LinkThis link leads to an external website and will open in a new tab and ask them to enable endpoint connections to s3.files.com. Until that is in place, connection attempts return the error SQL compilation error: Endpoint s3.files.com not allowed.

The bucket name for your Files.com site is your custom subdomain. If your custom subdomain is mysite.files.com, the bucket name is mysite.

The endpoint for your Files.com site is s3.files.com.

Use your Files.com S3-Compatible Endpoint credentials when configuring your Snowflake stage. For example:

CREATE STAGE my_files_dot_com_stage
  URL = 's3compat://mysite/path/to/folder/'
  ENDPOINT = 's3.files.com'
  CREDENTIALS = (AWS_KEY_ID = 'your_access_key_id' AWS_SECRET_KEY = 'your_secret_access_key')

Files.com's S3-compatible endpoint uses virtual-hosted-style requests, which is the format Snowflake requires. In virtual-hosted-style addressing, the bucket name is part of the subdomain (for example, mysite.s3.files.com) rather than part of the URL path.

Point the stage at the specific folder that contains your data files. Listing a folder returns only the files directly inside it, not inside any subfolders, so all files your Snowflake stage needs to access must be in a single flat folder on Files.com.

If your files are currently organized in subfolders, you can use a Copy Files automationExternal LinkThis link leads to an external website and will open in a new tab with the Flatten FoldersExternal LinkThis link leads to an external website and will open in a new tab option to consolidate them into a single flat folder that your Snowflake stage points at. Your original folder structure stays intact. The automation creates and maintains a separate flat copy for Snowflake to access. If files in different subfolders share the same name, they will overwrite each other in the destination.

Verify that Snowflake can successfully load and unload data filesExternal LinkThis link leads to an external website and will open in a new tab from your Files.com folder before putting the integration into production.

Running the Snowflake Test Suite

When you contact Snowflake support to enable connections to s3.files.com, they will ask you to run their public S3-compatible API test suiteExternal LinkThis link leads to an external website and will open in a new tab and share the results. The suite tests 9 S3 API operations and requires Java 1.8 or later and Maven to run.

Not all 9 tests pass, and not all of them need to pass for the integration to work. Many customers already use Snowflake with Files.com successfully, and passing all 9 is not a requirement for Snowflake to enable connections to s3.files.com.

Before running the suite, ensure the Files.com user associated with your S3-compatible API key has Full folder permissions on the test folder.

To run the suite:

  1. Clone the repository and build it following the test suite READMEExternal LinkThis link leads to an external website and will open in a new tab setup instructions, which require a GitHub access token for a build dependency.
  2. Set the required environment variables:
    • BUCKET_NAME_1 — your site's custom subdomain (for example, mysite)
    • REGION_1us-east-1 (Files.com ignores region; any value works)
    • REGION_2us-east-1 (use the same value as REGION_1)
    • S3COMPAT_ACCESS_KEY — your Files.com S3-compatible access key
    • S3COMPAT_SECRET_KEY — your Files.com S3-compatible secret key
    • END_POINTs3.files.com
    • NOT_ACCESSIBLE_BUCKET — any made-up subdomain (for example, notaccessible)
    • PREFIX_FOR_PAGE_LISTING — a dedicated empty folder path for testing (for example, test-prefix/). The test suite writes to this folder, so do not use a folder containing real data.
    • PAGE_LISTING_TOTAL_SIZE — the number of objects at that prefix. Create at least 1,001 objects in that folder before running the tests.
  3. Navigate into the s3compatapi directory and run mvn test -Dtest=S3CompatApiTest to execute all 9 tests.

The putObject test uploads files up to 5 GB, so the full suite can take over 2 minutes to complete.

The table below shows the expected result for each test.

TestResultNotes
getObjectMetadataPass
deleteObjectPass
generatePresignedUrlPass
getBucketLocationPartialBucket-region assertions pass. The nonexistent-bucket negative test fails because Files.com does not model per-bucket access semantics.
putObjectPartialBasic upload succeeds. The user-metadata subcase fails because x-amz-meta-* headers are not persisted.
getObjectPartialDownload and range-read assertions pass. The no-credentials subcase returns 400 instead of the expected 403.
listObjectsV2FailReturns 0 objects where the suite expects 2. The test suite uploads files several levels deep under the listing prefix; Files.com listing only returns objects at the immediate level of the prefix, not recursively.
deleteObjectsFailBlocked by the same listing behavior as listObjectsV2, returning 0 objects before reaching the delete assertion. Direct multi-delete works for addressable keys.
copyObjectFailThe destination parent path does not exist, which conflicts with Files.com's hierarchical storage.

Troubleshooting

Most issues come from incorrect folder permissions. Confirm that the S3-Compatible Endpoint access key Snowflake is using, and its associated user, have Full folder permissions on the target folder.

Confirm that Snowflake is configured to use s3.files.com as the endpoint URL. Custom domains are not supported for the endpoint URL.