A Labor-Market Data Provider Mounts Amazon S3 in Files.com to Eliminate Duplicate Dataset Uploads

A labor-market data provider sells one of the cleaner signals in alternative data. Its crawlers index job listings every day, directly from employers' own career pages, so a posting in its data closely tracks whether the job is actually live on the employer's site. That accuracy is why hedge funds, banks, central-bank research groups, and university economists buy the data to model hiring and the labor market.

What those clients buy is, physically, files. Daily datasets and custom market reports go out as CSV and compressed archives, in files that run from small up to tens and hundreds of gigabytes, with anything past a couple hundred gigabytes split before delivery. The provider decides how those files are built and where they are stored. It does not decide what its clients' systems can connect to. Before the provider mounted its bucket in Files.com, serving both needs meant uploading every dataset twice: once to S3 and again to the delivery layer clients could reach.

Every Dataset Uploaded Twice

The provider stored its datasets in its own Amazon S3 buckets, with Glacier behind them for archival. That was the right tier of record, and it never stopped being right. The clients were another matter.

The provider made Files.com its client-facing delivery layer. Clients connect on whatever their environment supports: FTP, FTP over explicit TLS, SFTP, WebDAV, or a browser, against the provider's own branded domain, each with their own credentials and folder. The provider's own application delivers files to logged-in customers through the Files.com API.

What that arrangement could not fix on its own was the double move. Every published file was built, then uploaded manually into S3 for storage, and uploaded again into the delivery layer for clients. At file sizes running into the hundreds of gigabytes, the duplicate step was a standing cost in every publishing cycle, and it fell on the engineers who ran the pipeline. The team had long wanted that step gone: if the platform could copy in and out of S3 directly, the provider could skip most of its uploading, and moving delivered content back into S3 for Glacier archival would get simpler too.

Neither the Bucket nor the Clients Would Move

The obvious fixes each gave up something the provider needed. Moving storage onto the delivery platform would abandon S3 and Glacier as the tier of record, and would pull the data away from the AWS environment where the files were increasingly being built. Writing an in-house synchronization pipeline between S3 and the delivery layer would trade a manual step for a system the provider had to build, run, and monitor. And the clients could not be changed at all: a bank's transfer tooling is the bank's decision, and the provider's business is meeting each client on whatever protocol that client already has.

As the build pipeline moved off the provider's own datacenter and into AWS, the round trip got harder to justify. Files were generated in AWS and stored in AWS, then uploaded out again so clients could reach them. What the provider needed was a delivery surface that presented S3 content directly, with no migration, no second copy, and no sync machinery of its own to maintain.

Mounting the Bucket Instead of Copying It

The provider set exactly that up. Using Files.com's Remote Server Mount, the team mounted an S3 bucket from its own AWS account onto a folder in its Files.com site. Files.com became a window onto the bucket rather than a second home for its contents.

The mount works in real time. A file the pipeline writes into S3 is, from that moment, a file in the Files.com folder tree. No upload job runs, and no second copy exists.

Everything the provider had already built on Files.com applies to that content unchanged. The mounted files sit behind the same per-client folders and credentials, travel over the same protocols, appear under the provider's own domain, and reach clients through the same API-driven delivery inside the provider's application. And S3 stayed exactly what it was: the provider's storage and archival tier of record, with Glacier behind it, in the provider's own AWS account.

The Second Upload Is Gone

With the mount in production, the provider replaced a publishing cycle that moved every dataset twice with one that moves it once, into the bucket it already owned. The provider went on to manage tens of terabytes on the platform, with single files running to hundreds of gigabytes.

The change also decoupled delivery from infrastructure. When the provider later moved the upload side of its pipeline fully onto AWS EC2, the Files.com integration carried across the re-platforming, and the folders, credentials, and protocols its clients connect to did not move. Where files are built and where they are stored are now decisions the provider makes for its pipeline, without touching the surface its clients depend on.

One Copy, Every Protocol

Since the mount went live, publishing at the provider ends where storage begins. The pipeline writes a dataset into the company's own bucket, and that single write is the delivery.

The provider's setup shows that those two decisions never had to be traded against each other. The data stayed in S3, chosen for the pipeline and the archive. The clients stayed on FTP, SFTP, and the browser, chosen by their own IT. Files.com is the layer that lets both stand: the storage the provider already runs, made reachable by every client it serves.