Unless you are testing a file utility class, if you access files during your test, you exceeding the scope of the unit that you should be testing
That statement sums up the principle behind this rule. However, there are more pragmatic reasons to consider as well.
Clarity / Evident Data
Often the motivation for accessing files from a unit test is to read test data upon which the class under test depends. Separating test data from the code of your test makes that test significantly harder to understand and debug. Kent Beck refers to this unit testing best practice as Evident Data. Show the reader of your test the intent of the code being tested by using clear and understandable data as parameters. Once you move that data into a separate file, especiall in binary formats, the maintenance cost and usefulness of that test becomes significantly worse.
Speed Of Execution
Reading files is much slower than executing code.
Ease of Setup and Brittleness Issues
Unit tests that depend on files being in the right place in order to pass