strange behavior while writing to %appdata% in python

Question

I was trying to write some data to %appdata%. All seemed to work like shown in the output of Script1. The new directories are being created and the file is saved and the data gets retrieved successfully as well. But trying to look at the data in file explorer, the folder wasn't there! CMD couldn't find the file and directory either.

Later I created the file manually and checked, what happened. The CMD could now find the file (which I just manually created), but when trying to read the file with python it'd output me the python ghost file contents test data 123 and not what I've just written into it! (I also double-checked with WSL that the new file actually contains test data 456.)

1. What's going on?
2. Is it an issue with my windows or python installation?
3. Where is the python ghost version of the file being stored?
4. How to resolve the issue?

Script1 (Creating the file with test data 123):

import os
import subprocess


appdata        = os.getenv('APPDATA')
directory_path = f"{appdata}\\com-company\\prod-product-version3"
file_path      = directory_path + "\\file1.txt"


print(f"Directories Exist: {os.path.exists(directory_path)}")
if not os.path.exists(directory_path):
    os.makedirs(directory_path)
    print("Directories created")
print(f"Directories Exist: {os.path.exists(directory_path)}")



print(f"File Exist: {os.path.exists(file_path)}")
print(f"Writing File: {file_path}")
with open(file_path, 'w')as fp:
    fp.write("test data 123")
print(f"File Exist: {os.path.exists(file_path)}")


print(f"Reading File: {file_path}")
with open(file_path, 'r')as fp:
    print(f"File Content: {fp.read()}")



print('---------------------')
cmd = f"dir {directory_path}"
try:
    output = subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT, text=True)
    print(output)
except subprocess.CalledProcessError as e:
    print(f'Error: {e}')
    print(f'Error message:\n{e.output}')


print('---------------------')
cmd = f"dir {file_path}"
try:
    output = subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT, text=True)
    print(output)
except subprocess.CalledProcessError as e:
    print(f'Error: {e}')
    print(f'Error message:\n{e.output}')

Output:

Directories Exist: False
Directories created
Directories Exist: True
File Exist: False
Writing File: C:\Users\one\AppData\Roaming\com-company\prod-product-version3\file1.txt
File Exist: True
Reading File: C:\Users\one\AppData\Roaming\com-company\prod-product-version3\file1.txt
File Content: test data 123
---------------------
Error: Command 'dir C:\Users\one\AppData\Roaming\com-company\prod-product-version3' returned non-zero exit status 1.
Error message:
The system cannot find the file specified.

---------------------
Error: Command 'dir C:\Users\one\AppData\Roaming\com-company\prod-product-version3\file1.txt' returned non-zero exit status 1.
Error message:
The system cannot find the path specified.

Creating C:\Users\one\AppData\Roaming\com-company\prod-product-version3\file1.txt manually and writing data into it:

test data 456

Script2 (reading test data 123 even though it contains test data 456):

import os
appdata        = os.getenv('APPDATA')
directory_path = f"{appdata}\\com-company\\prod-product-version3"
file_path      = directory_path + "\\file1.txt"

print(f"File Exist: {os.path.exists(file_path)}")

print(f"Reading File: {file_path}")
with open(file_path, 'r')as fp:
    print(f"File Content: {fp.read()}")

Output:

File Exist: True
Reading File: C:\Users\one\AppData\Roaming\com-company\prod-product-version3\file1.txt
File Content: test data 123

Double checking with WSL:

cat /mnt/c/Users/one/AppData/Roaming/com-company/prod-product-version3/file1.txt
Output: test data 456

PS: I rebooted my system and python still thinks the file contains test data 123. And writing normally works just fine:

with open('C:\\Users\\one\\Desktop\\file2.txt', 'w') as fp:
    fp.write('test data 789')

Answer 1

This is a bug in the Windows Store version of Python.

It's still present, even with Python 3.12 installed from the Microsoft Store. Using procmon and some blackbox testing I found the following:

Any paths under C:\Users\username\AppData\REST-OF-THE-PATH are actually treated as C:\Users\username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\REST-OF-THE-PATH.

This happens regardless of working in a venv or not, and regardless of the library call you're using (i.e. pathlib and open behave the same)

Answer 2

You might be hitting a cache issue.

Try refreshenv, it should refresh the APPDATA variable that you are using to construct the file path.

python -m site --user-site can show you where python stores site-packages dir where python might be caching files/modules.

Using a function that operates on a lower-level might be helpful, like os.scandir