Bates White Automates 100 GB Databricks-to-File-Server Transfers With Files.com

Bates White is an economic consulting firm headquartered in Washington, DC. Its economists provide analysis and expert testimony to law firms, Fortune 500 companies, and government agencies, in matters running from antitrust enforcement actions to mass tort bankruptcies. Global Competition Review lists it among the world's top economics firms in competition policy, and its mass torts practice has spent more than 25 years building proprietary models and claims databases; in one asbestos bankruptcy, the presiding judge called a claims database the firm built the most extensive of its kind produced to date.

That is the shape of the work: expert opinion built on top of very large empirical datasets. The computation happens in the cloud. Every project begins in Databricks, writing its output to Amazon S3. The analysis happens on the ground. The economists and data scientists who turn that output into results run their scripts against the firm's own network file server, where datasets of 100 GB to 1 TB move at local-network speed. A firm built this way faces the same question on every single project: how does the output get from the bucket to the server?

Every Project Started in Databricks, and Every File Moved by Hand

For years, the answer was a person. For small files, a data engineer opened WinSCP, browsed the bucket, and downloaded files to the network share by hand. But WinSCP could not handle the files this work actually produces. Individual files reach 100 GB, and anything much past 10 GB defeated it. Large copies fell back to the AWS CLI: a command composed by hand for each transfer, with the VPC endpoint typed in every time to route the copy over the firm's 5 Gbps direct connect to AWS. Over the ordinary internet path, a 100 GB file took about an hour to come down.

There was no automated way to do any of it. And a pulled file immediately began to age. When the copy in S3 was updated, nothing updated the copy on the file server, so version control between the bucket and the local network became a problem of its own.

The cost repeated on every engagement, because every engagement starts this way. Data engineers carried a per-transfer procedure as standing overhead: notice the new files, pull them, place them. Analysts either waited for someone to run it, or worked from a local copy nobody could certify as current. Bates White had a pipeline problem being solved with hand labor: the seam between cloud computation and local analysis, crossed manually on every project.

What an Automated Path Had to Do

The obvious fix, teaching the analysts to pull from S3 themselves, was the problem restated. The point of the file server is that it is where analysts already work. Their scripts read from it and write derived files back to it, and asking those people to also manage bucket credentials, endpoint parameters, and CLI syntax defeats the purpose. Desktop transfer tools had already failed at these file sizes. And with multiple people updating buckets across active projects, manual synchronization could not be trusted to keep local copies current.

So the fix had to be a pipeline, and a specific kind of pipeline: one that connects to S3 on one side and lands files on the on-prem file server on the other, moves 100 GB files as routinely as small ones, carries updates as well as new files, and asks nothing of the people who consume the output. It could be relaxed about speed. No downstream system waits on these files; people use them, and the data engineering team judged a 15-minute lag between updates entirely acceptable.

Bates White selected Files.com to be that pipeline.

S3 on One Side, the Files.com Agent Inside the Network on the Other

Files.com became the standing bridge between the firm's cloud analytics storage and its own network, and the build was configuration rather than code.

Bates White connected the S3 folder where Databricks writes its results to Files.com as a remote server. Inside Bates White's network, the Files.com Agent provided a path onto the file server using only outbound connections, requiring no inbound firewall rule and exposing nothing to the internet. Nothing upstream changed: Databricks jobs keep writing to S3 exactly as they always did.

Between the two, a Files.com sync moves new and updated files from the bucket, through the Agent, onto the network file server. Updated matters as much as new: when a file changes in S3, the sync carries the change down, which is what closes the version-control gap. The flow stays one-directional by design. One population produces the files, writing from Databricks into S3. The other consumes them from the local share, and the derived files analysts create stay in local folders that never need to travel back.

A Standing Pipeline Instead of a Per-Transfer Procedure

With the workflow tested and in production, Bates White replaced a hand-operated transfer procedure with a pipeline that runs on its own.

• Databricks output lands on the file server where analysts already work. Nobody browses a bucket in WinSCP, and nobody composes a CLI command with a hand-typed VPC endpoint to move a file.
• The size ceiling is gone from this path. A 100 GB file moves through the same automated flow as a small one, instead of forcing a fallback procedure.
• Local copies track the bucket. An analyst's script reads the current version of a file, not whichever copy someone last pulled, and the sync cadence sits comfortably inside the 15 minutes the team asked for.
• The next bucket is configuration, not a build. Bringing another project bucket into the flow requires a remote server connection and a sync, following a pattern already proven in production.

Where Cloud Output Simply Arrives

A data engineer finishes a job in Databricks, and within minutes the output is sitting on the network file server. An analyst opens the share and the files are there, current and ready for local scripts, with no one having moved them by hand.