How Madison AI Moved 1.8 Million Files Off SharePoint Without Rebuilding Its Government AI Pipeline

Madison AI builds private AI assistants for US local governments. Launched in 2024, the company serves roughly 50 city and county clients, delivering purpose-built assistants for the functions a local government actually runs: the city manager's office, planning and community development, HR, legal. Each assistant is built entirely from that government's own records. Meeting minutes, staff reports, zoning codes, master plans, and internal policies become a closed-loop, retrieval-based knowledge base that staff query instead of digging through archives.

That proposition puts a serious file operation at the center of the business. Every engagement begins with ingesting a client's record library: scraping, OCR, conversion, and sanitization run on an on-premises Synology NAS, and the converted output lands in that client's own Azure Blob Storage containers, where Azure AI Search builds the indexes the retrieval system reads. An assistant's answers are only as trustworthy as the pipeline that carried the records into it. And between the local processing tier and the per-client cloud containers, everything passed through a staging layer.

Two Document Libraries, 1.8 Million Files, and SharePoint's Real Ceiling

That staging layer was SharePoint: two document libraries holding client data for the whole customer base, 1.8 million files and roughly 7 terabytes.

“If you hit 300,000 files, it effectively stops working. The more you go over 300,000 in a site, the worse it gets, and we're at about 1.8 million files at this point.”

Six times past the practical ceiling, the estate was failing in every direction at once. Automated processes were throttled. The large libraries could no longer sync at all. The throttling cascaded beyond the libraries themselves: OneNote notebooks elsewhere in the tenant, with no connection to client data, started failing too. Worse, files blipped out of the libraries and back, and some were lost outright.

For a company whose product is AI built from a government's complete records, that last failure was the one that mattered.

“We need to trust it, and if we can't fully, then how are our clients going to?”

The estate was also expensive and labor-intensive to keep alive. Additional SharePoint storage expense kept climbing, and the limit had to be raised twice inside six weeks as new client data arrived. Moving processed files out of SharePoint and into each customer's Blob container depended on hand-rolled Microsoft Graph API scripts running on Windows Task Scheduler, three to four streams at a time, monitored by hand. Only two engineers could run or control that path.

Why It Couldn't Be Fixed in Place

The problem could not be configured away. SharePoint's file ceiling was architectural, and client growth kept pushing past it: Madison AI was ingesting roughly a terabyte of new records a month, with new city and county clients closing continuously.

The obvious alternatives had already failed. Transferring files directly from the NAS to Azure Blob had been tried and abandoned after network errors caused file-deletion incidents. And the backend of the product could not simply be rebuilt around new storage: the Azure AI Search indexers that powered every client's assistant were bound to the existing per-customer Blob containers, and retying them across roughly 50 clients and 80 to 100 separate data lakes would have meant reworking the foundation of the product itself.

So whatever replaced SharePoint had a specific job. It had to carry millions of files with no practical ceiling. It had to reach the NAS, because the OCR and conversion tooling required files native on that machine. It had to deliver converted files into each client's own Blob container on a schedule, with no scripts to babysit. It had to keep every government's data in its own scoped folder tree. And it had to take over with nothing missed, while ingestion kept running.

Madison AI selected Files.com to be that orchestration layer.

Files.com Between the NAS and Every Client's Blob Container

Files.com became the layer that presents the whole estate as one set of governed folders and runs the movement between its parts on its own.

On the local side, the Files.com Agent runs as a Docker container directly on the Synology NAS, connecting over an outbound-only connection. The NAS stays, because the conversion path needs local files, and its two client libraries now appear as ordinary Files.com folders. Internal delivery and audit staff work on them in the browser at a scale SharePoint could never serve.

On the cloud side, every customer's containers are connected to Files.com as remote servers: 133 Azure Blob connections, roughly two per client, one holding converted and indexed documents and one holding the unconverted source files the application displays and cites. Forty-nine scheduled Files.com Syncs push converted output into the container each client's Azure AI Search indexer reads. The indexers never moved.

Access is scoped with folder-level permissions that mirror each government's own organization, keeping each client isolated while giving departments the appropriate reach. A city manager's scope covers everything for that client; HR and legal reach only their own folders.

Sarah Orner owned the implementation through roughly weekly guided onboarding sessions with a Files.com onboarding architect, doing the configuration between sessions. The Agent was configured the day the team first attempted it; Orner then built a test agency to validate file movement in both directions before setting up the first Sync. The existing Azure Blob remote servers were connected during onboarding, and the pattern then expanded across customer containers. Initial onboarding ended with Orner self-sufficient on the configuration.

The cutover protected the data the whole way. Files were copied out of SharePoint rather than moved, so SharePoint stayed intact as a fallback while the two systems ran in parallel for a handful of months. One client's flow went live first, then the pattern fanned out and became the standard for every new client. Preserving the full folder structure with nothing missed was the stated bar, and it held.

Unattended Syncs Without SharePoint Storage Overhead

With Files.com in production, Madison AI replaced a degrading SharePoint estate and a two-engineer script path with a file layer that runs itself.

• After the initial full syncs carried the load, every client sync runs on schedule with nobody watching it. The Microsoft Graph scripts and Task Scheduler jobs that only two engineers could run and monitor are retired.
• The additional SharePoint storage expense—and the repeated limit increases—stopped when the libraries left SharePoint.
• The completeness doubt went with it. Client records now sit on a layer with no file ceiling, absorbing roughly a terabyte of new records every month.
• Onboarding the next government client is now a repeat of the same pattern: a scoped folder tree, a pair of Blob connections, and a Sync. The file layer is no longer the constraint on that growth.

When the automated syncing of indexed files first came online, Sarah Orner's reaction summed up what the scripts had been costing:

“You're going to save me hours. I'm so excited.”

The Pipeline Stayed, the Layer Underneath It Changed

The larger lesson is what Madison AI did not do. It never rebuilt its AI pipeline to fix its file problem. The Azure AI Search indexers stayed bound to the storage they already read, the NAS kept doing the local work only it can do, and Files.com replaced the failing layer between them while old and new ran side by side until the new one had proved itself. The file layer feeding a production RAG system turned out to be swappable, and Madison AI swapped it under a business that never stopped ingesting.