LinkUp Mounts Amazon S3 in Files.com to Eliminate Duplicate Dataset Uploads

LinkUp, a Minneapolis-based labor-market data company and now a division of GlobalData plc, sells one of the cleaner signals in alternative data. Its crawlers index job listings every day, directly from the career pages of more than 80,000 employers rather than from aggregators, so a posting in LinkUp's data closely tracks whether the job is actually live on the employer's own site. That accuracy is why hedge funds, banks, central-bank research groups, and university economists buy the data to model hiring and the labor market.

What those clients buy is, physically, files. Daily datasets and custom market reports go out as CSV and compressed archives, in files that run from small up to tens and hundreds of gigabytes, with anything past a couple hundred gigabytes split before delivery. LinkUp decides how those files are built and where they are stored. It does not decide what its clients' systems can connect to. Before LinkUp mounted its bucket in Files.com, serving both needs meant uploading every dataset twice: once to S3 and again to the delivery layer clients could reach.

Every Dataset Uploaded Twice

LinkUp stored its datasets in its own Amazon S3 buckets, with Glacier behind them for archival. That was the right tier of record, and it never stopped being right. The clients were another matter.

LinkUp made Files.com its client-facing delivery layer. Clients connect on whatever their environment supports: FTP, FTP over explicit TLS, SFTP, WebDAV, or a browser, against LinkUp's own branded domain, each with their own credentials and folder. LinkUp's own application delivers files to logged-in customers through the Files.com API.

What that arrangement could not fix on its own was the double move. Every published file was built, then uploaded manually into S3 for storage, and uploaded again into the delivery layer for clients. At file sizes running into the hundreds of gigabytes, the duplicate step was a standing cost in every publishing cycle, and it fell on the engineers who ran the pipeline. Cross named the step he wanted gone as early as 2017: if the platform could copy in and out of S3 directly, LinkUp could skip most of its uploading, and moving delivered content back into S3 for Glacier archival would get simpler too.

Neither the Bucket nor the Clients Would Move

The obvious fixes each gave up something LinkUp needed. Moving storage onto the delivery platform would abandon S3 and Glacier as the tier of record, and would pull the data away from the AWS environment where the files were increasingly being built. Writing an in-house synchronization pipeline between S3 and the delivery layer would trade a manual step for a system LinkUp had to build, run, and monitor. And the clients could not be changed at all: a bank's transfer tooling is the bank's decision, and LinkUp's business is meeting each client on whatever protocol that client already has.

As the build pipeline moved off LinkUp's own datacenter and into AWS, the round trip got harder to justify. Files were generated in AWS and stored in AWS, then uploaded out again so clients could reach them. What LinkUp needed was a delivery surface that presented S3 content directly, with no migration, no second copy, and no sync machinery of its own to maintain.

Mounting the Bucket Instead of Copying It

In 2021, LinkUp set exactly that up. Using Files.com's Remote Server Mount, the team mounted an S3 bucket from its own AWS account onto a folder in its Files.com site. Files.com became a window onto the bucket rather than a second home for its contents.

The mount works in real time. A file the pipeline writes into S3 is, from that moment, a file in the Files.com folder tree. No upload job runs, and no second copy exists.

Everything LinkUp had already built on Files.com applies to that content unchanged. The mounted files sit behind the same per-client folders and credentials, travel over the same protocols, appear under LinkUp's own domain, and reach clients through the same API-driven delivery inside LinkUp's application. And S3 stayed exactly what it was: LinkUp's storage and archival tier of record, with Glacier behind it, in LinkUp's own AWS account.

The Second Upload Is Gone

With the mount in production, LinkUp replaced a publishing cycle that moved every dataset twice with one that moves it once, into the bucket it already owned. By 2022, LinkUp had more than 28 TB under active management on the platform, with single files running to hundreds of gigabytes.

The change also decoupled delivery from infrastructure. When LinkUp later moved the upload side of its pipeline fully onto AWS EC2, the Files.com integration carried across the re-platforming, and the folders, credentials, and protocols its clients connect to did not move. Where files are built and where they are stored are now decisions LinkUp makes for its pipeline, without touching the surface its clients depend on.

One Copy, Every Protocol

Since the mount went live, publishing at LinkUp ends where storage begins. The pipeline writes a dataset into the company's own bucket, and that single write is the delivery.

LinkUp's setup shows that those two decisions never had to be traded against each other. The data stayed in S3, chosen for the pipeline and the archive. The clients stayed on FTP, SFTP, and the browser, chosen by their own IT. Files.com is the layer that lets both stand: the storage LinkUp already runs, made reachable by every client it serves.