Text Encoding

If names arrive with garbled accents, a first column cannot be found, or an extra character appears at the end of every record, the problem may be how the file's bytes are interpreted. A file holds bytes; a character encoding defines how those bytes represent text. Copying the bytes faithfully preserves the source encoding, including one the receiver did not expect.

Bytes and Characters

UTF-8 represents Unicode text, including characters from many writing systems. Older exports may use a more limited encoding, such as Windows-1252. The same visible character has different bytes in those encodings: é is C3 A9 in UTF-8 and E9 in Windows-1252. ASCII letters and digits look the same in both. An export containing only those characters can read correctly in either encoding; check the configured encoding before processing names with accents.

Reading UTF-8 bytes as Windows-1252 can turn é into é. Reading bytes that are invalid in the selected encoding can instead produce an error or a replacement character such as �. Saving that replacement can discard information, so investigate using the original file rather than repeatedly opening and saving the garbled version.

The strongest evidence is the exporting system's configured encoding. A text editor that can reopen a file with an explicit encoding helps compare that setting with the contents. Automatic detection is a guess when several encodings can interpret the same bytes. Converting text means decoding with the source encoding and then writing with the destination encoding, rather than merely changing the encoding label.

In TransformScript, input and output encoding are separate choices. Text input defaults to UTF-8; an explicit input encoding tells the reader how to decode the source bytes. The output encoding controls the new file. Changing the output setting alone cannot repair text that was read incorrectly.

Byte Order Marks

A byte order mark (BOM) is a signature at the beginning of a Unicode text file. UTF-16 can use it to identify byte order. In UTF-8 it is the three-byte sequence EF BB BF, identifying the encoding rather than changing byte order. UTF-8 text can also exist without this signature.

Whether a BOM helps depends on the receiving reader. Excel can use one to recognize a UTF-8 CSV. A reader that treats the signature as ordinary text can include an invisible character in the first header, so a name that looks like CustomerID differs from the expected field name. A reader may also reject a signature before a quoted first field.

For CSV input to TransformScript, supply UTF-8 without a BOM or remove the signature before CSV parsing. Its reader preserves the signature, even with UTF-8 selected as the input encoding.

Export tools and editors offer different encoding choices. A UTF-8 export with a BOM and a UTF-8 export without one contain the same characters after the signature but are different byte sequences. PowerShell's output defaults also differ by version and command. Specify the receiver's BOM requirement alongside the encoding, and include the first header in a representative import test. Files.com's bulk CSV importer documents its own handling.

Line Endings

Text files encode the end of a line with control characters. Windows-oriented tools commonly write carriage return followed by line feed (CRLF, \r\n); Linux and modern macOS tools commonly write line feed alone (LF, \n). The receiving application determines which it accepts.

A reader that splits only on LF may leave the preceding CR attached to the final field of each CRLF record. A reader expecting CRLF may treat LF-only input as one long record. A text editor's line-ending display or a view of the bytes distinguishes these cases without changing the file.

Record boundaries also depend on the format. A quoted CSV field can contain a line break, so splitting text at every newline is not a CSV parser. Similarly, changing all line breaks can change the contents of a field as well as the separators between records.

TransformScript's CSV writer defaults to LF and can produce CRLF through its output properties. This is separate from FTP ASCII transfer mode, which performs line-ending conversion during transfer. Binary FTP mode preserves the bytes and lets a defined processing step own any required conversion.

Unicode Normalization

Even correctly decoded Unicode text can represent the same visible spelling in different ways. For example, an accented é can be one character or an e followed by a combining acute accent. Software comparing the sequences exactly can treat those spellings as different identifiers or filenames.

Unicode normalization gives equivalent sequences a consistent representation. NFC favors composed forms where available; NFD decomposes them. Both can be encoded in UTF-8, so selecting UTF-8 alone does not resolve the difference. Text produced or saved on different systems can use different forms. Check the file and the producing application's normalization behavior when text comparisons differ.

For values inside a dataset, any normalization rule belongs in the sender and receiver's data specification. Files.com's filename comparison is a separate product rule: it preserves supplied name spelling while using its own map to compare paths. Ordinary NFC or NFD normalization does not reproduce that map or establish how an external system compares names.