Databricks

Databricks is a unified data analytics platform built on Apache Spark that supports data engineering, machine learning, and collaborative data science at scale.

Files.com integrates with Databricks through the Files.com S3-Compatible Endpoint. You configure a Databricks cluster to treat your Files.com site as an S3-compatible store, and Spark reads and writes data directly from your Files.com folders using the s3a:// connector.

This integration provides cluster-scoped S3A access to your Files.com site. It is not governed by Unity Catalog. Unity Catalog external storage relies on cloud-provider IAM role assumption, which is not part of the S3-compatible credential model. If your organization requires Unity Catalog governance over data exchanged with Files.com, see Governance Considerations below.

Configuring Files.com

Your Files.com site includes an S3-Compatible Endpoint. To connect Databricks, you need an access key and a user with the right folder permissions.

1. Generate an S3-compatible API Key. This produces the Access Key ID and Secret Access Key that Databricks will use. The key must be associated with a user on your site.
2. Configure folder permissions for the folder you want Databricks to access so that the key's associated user has access. Read permission is the minimum required for Databricks to load data. Write permission is required if Databricks will also write results back to Files.com.
3. Before configuring Databricks, sign in as the key's user and confirm the user can see the expected folder and files.

Configuring Databricks

Databricks connects to Files.com using Spark's S3A connector. You can configure the connection at the cluster level (applies to all notebooks on the cluster) or at the notebook/session level (useful for testing before committing to a cluster config change).

Cluster Access Mode

Custom S3A credentials only take effect on clusters where Spark configuration is not overridden by Unity Catalog's credential injection. Use a cluster with Single user (assigned) access mode, or a no-isolation cluster. On a standard Unity Catalog shared cluster, these settings may be silently ignored or conflict with UC-managed credentials.

Connection Settings

Use the following values from your Files.com S3-Compatible Endpoint:

Files.com presents a single bucket that maps to the top level of your site. Folders inside your site appear as key prefixes, for example s3a://default/reports/2026/data.csv or s3a://mysite/reports/2026/data.csv.

Cluster Spark Configuration

This option applies the connection to every notebook on the cluster and persists across restarts. Add the following to the Spark config in the Advanced options for your cluster.

spark.hadoop.fs.s3a.endpoint https://s3.files.com
spark.hadoop.fs.s3a.access.key {{secrets/files_com/access_key}}
spark.hadoop.fs.s3a.secret.key {{secrets/files_com/secret_key}}
spark.hadoop.fs.s3a.path.style.access true
spark.hadoop.fs.s3a.connection.ssl.enabled true
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem

Do not paste your Secret Access Key into the Spark config in plaintext. It is visible to anyone with edit access to the cluster. Store credentials in a Databricks secret scope and reference them with the {{secrets/scope/key}} syntax shown above.

Notebook (Per-Session) Configuration

This option is useful for testing without changing cluster settings. Set the same values from within a notebook cell:

spark.conf.set("fs.s3a.endpoint", "https://s3.files.com")
spark.conf.set("fs.s3a.access.key", dbutils.secrets.get("files_com", "access_key"))
spark.conf.set("fs.s3a.secret.key", dbutils.secrets.get("files_com", "secret_key"))
spark.conf.set("fs.s3a.path.style.access", "true")
spark.conf.set("fs.s3a.connection.ssl.enabled", "true")

Accessing Data

Reference files using the s3a:// scheme with your bucket name and folder path as the key prefix:

# List a folder
dbutils.fs.ls("s3a://default/path/to/folder/")

# Read a file
df = spark.read.csv("s3a://default/path/to/folder/myfile.csv", header=True)
df.show()

# Write results back (requires Write permission on the folder)
df.write.mode("overwrite").parquet("s3a://default/path/to/output/")

Verifying the Integration

Before putting the integration into production, confirm that Databricks can read from and (if needed) write to your Files.com folder.

1. List — run dbutils.fs.ls("s3a://default/your/folder/") and confirm the expected files appear.
2. Read — load a known file and inspect it with .show() or .count().
3. Write (if used) — write a small test DataFrame to a scratch folder, then confirm the file appears in the Files.com web interface.

Governance Considerations

This integration gives Databricks direct, cluster-scoped access to your Files.com site. It does not place the data under Unity Catalog governance, lineage, or access control. Access is governed entirely by Files.com folder permissions on the key's associated user.

Organizations that require Unity Catalog governance over data exchanged with Files.com typically use Files.com as the transfer and orchestration layer, moving files into and out of the cloud object storage that Unity Catalog already governs, rather than as a Unity Catalog external location itself. Contact Files.com support if you would like to discuss the right architecture for your environment.

Troubleshooting Databricks Connections

Most issues come from incorrect access permissions. Confirm that the S3-Compatible Endpoint access key Databricks is using, and its associated user, have the correct permissions on the target folder.

Bucket Not Found or DNS Errors

Path-style addressing is not enabled. Set fs.s3a.path.style.access to true.

Credentials Appear Ignored, or Unity Catalog Errors

The cluster is in Unity Catalog shared access mode. Use a Single user (assigned) or no-isolation cluster instead.

Access Denied or 403

The key's user lacks permission on the target folder. Confirm Read permission (and Write, for unloads) is set, and that permissions have not changed since setup.

Signing or Region Errors

A region value is required even though Files.com ignores it. Set fs.s3a.region (or aws.region) to any value, e.g. us-east-1.

TLS or Certificate Errors

Confirm fs.s3a.connection.ssl.enabled is true and the endpoint URL is correct. Files.com serves a valid public certificate; no custom truststore is required.

Wrong Path or Empty Listing

Confirm you are using default (or your custom subdomain) as the bucket and the correct folder prefix, e.g. s3a://default/folder/file.csv.