Unless you are unit testing a file utility class, if you access files during your test, you exceeding the scope of unit that you should be testing.
However, there are more pragmatic reasons to consider as well.
Clarity / Evident Data
Many times, the motivation for accessing files is to read test data upon which the class under test depends. Separating test data from the code of your test makes that test significantly harder to understand and debug. Kent Beck refers to this unit testing best practice as Evident Data. Show the reader of your test the intent of the code being tested by using clear and understandable data as parameters. Once you move that data into a separate file, especiall in binary formats, the maintenance cost and usefulness of that test becomes significantly worse.
Speed of execution
If Ashcroft fails your unit test because some class was reading or writing to a file, you will be in one of the following cases: