Log File Streaming

Log File Streaming writes Files.com audit and activity logs to files on your site at a configured interval. Logs are delivered as JSON or CSV files and stored in selected destination folders. This approach is intended for workflows that rely on file-based access, scheduled processing, long-term retention, or offline analysis.

For workflows that require real-time monitoring and alerting within a SIEM platform, Files.com offers separate SIEM integrations. These integrations deliver audit and activity logs directly to supported SIEM platforms as events occur, rather than writing them to files on a defined interval.

How Log File Streaming Works

When Log File Streaming is enabled, Files.com collects the selected log types during each configured output interval. At the end of the interval, all collected entries are written to one or more files in the destination folder. The destination folder can be a folder on Files.com or a folder backed by a configured Remote Server. Each file contains all log entries generated for a specific log type during that time window.

Files are written once at the end of each interval and are not modified again by Files.com. Any subsequent activity is written to new files in later intervals. This allows files to be processed safely as soon as they appear, without concern for partial records or changing data.

Downstream systems often detect new log files by polling. Because files are created at a predictable interval, many workflows periodically list the contents of the destination folder using file access methods or APIs, and process files based on their timestamps or naming patterns.

For event-driven processing, Files.com Webhooks can be used to notify external systems when new files are created.

Log File Streaming Configuration

A log file stream is configured by providing a name, selecting destination folders, choosing a file format, setting an output interval, and selecting the log types to include.

Destination folders determine where log files are written and how they can be accessed, retained, or processed. Destination folders can be folders on Files.com or folders on Remote Server Mounts. Using remote server mounts is appropriate when logs need to be stored outside the Files.com site, such as when logs are processed by external analytics pipelines or when they must reside in a specific storage environment to meet internal retention or access requirements.

Each log type can be configured to write to its own destination folder, or multiple log types can be written to a single folder. This provides flexibility in how logs are organized, depending on how they will be accessed, ingested, or processed downstream.

The output interval controls how often log files are created and can be set between 5 and 360 minutes. By default, the interval is 60 minutes. For example, when the interval is set to 60 minutes, log files are written every 60 minutes and include all events generated for each selected log type during the preceding 60-minute window. Shorter intervals produce smaller, more frequent files, while longer intervals produce larger files containing more entries.

File Formats and Naming

Log File Streaming supports JSON and CSV formats. The same log data is written regardless of format, and the choice depends on how the files will be consumed.

Log files follow a consistent naming pattern in the form [LogType]_[Timestamp].[extension]. The timestamp represents when the file was created and can be used to order files chronologically. Including the log type in the file name allows different categories of logs to be processed independently. File extensions reflect the selected format, either .json or .csv.

When a Log File Stream is created or updated and then enabled, a small test file is written to the selected destination folder. This file is used to validate that the configuration is correct and that the destination folder is accessible. Subsequent files contain actual log data generated during each output interval.

Batching and Delivery

Log files are written at the end of each configured output interval. For a given log type, all log entries generated during that interval are written to a single file. For example, if the interval is set to 60 minutes, one file is created at the end of the 60-minute window containing all events generated for that log type during that period.

At the end of the next interval, a new file is created containing all log entries generated since the previous file was written. Files are written as complete outputs and are not modified after creation. Files are not compressed and contain only complete log records.

Supported Log Types

Log File Streaming supports the following log types. Only selected log types are written to files.

Use Cases

Log File Streaming is used when audit and activity logs need to be written, stored, or processed as files rather than consumed through real-time streaming. File-based delivery provides fixed file boundaries and supports batch-oriented workflows that rely on predictable file creation.

Data Warehousing and Analytics Pipelines

Log File Streaming is used to ingest audit and file activity data into data warehouse and analytics systems such as Snowflake or BigQuery. These systems typically ingest data from batch files rather than continuous streams. Logs written as JSON or CSV files can be processed incrementally by reading completed files from a folder based on timestamps or file names. This approach avoids the need for real-time collectors and allows ingestion jobs to run on a defined schedule.

Compliance Audits and Historical Analysis

File-based logs are used for compliance and audit workflows that require durable, immutable records of activity. Logs written to files can be retained for defined periods, reviewed during audits, and provided as records when required. Because files are created at fixed intervals, activity can be examined for specific time windows such as hourly or daily periods without querying live systems.

Long-Term Retention and Archival

Logs are often retained for extended periods to meet internal or regulatory requirements. Log File Streaming allows logs to be written to folders that support long-term retention policies. Files can be archived, retained, or moved according to organizational requirements while remaining accessible for later review.

Integration with External Processing Systems

Log File Streaming is used when logs need to be consumed by external systems that operate on files. Logs can be written to locations where downstream processing jobs, transformation workflows, or replication processes already read data. This allows external systems to process log files directly without retrieving them from Files.com.

Environments with Restricted Network Access

In environments with restricted outbound connectivity or controlled network access, file-based delivery avoids the need for continuous outbound connections. Logs remain within approved storage locations and can be accessed or transferred using permitted file access methods.

Manual Review and Troubleshooting

File-based logs support manual inspection and troubleshooting. CSV files can be opened in spreadsheet tools, while JSON files can be examined using standard text or analysis tools. This is commonly used to investigate specific time periods, validate system behavior, or troubleshoot issues such as failed transfers or configuration changes.

Legal and Forensic Review

Log File Streaming is used in legal and forensic scenarios that require static, verifiable records of activity. Files generated for specific intervals can be preserved as records and stored securely. These files provide an audit trail that supports investigations, incident analysis, and chain-of-custody requirements.