Python Libs – Why “profile” can’t be context manager?

Why Python standard library “profile” can’t be context manager like this

 

Zero, what is “profile” module in Python? According to Python 3 Documentation:

profile, a pure Python module whose interface is imitated by cProfile, but which adds significant overhead to profiled programs. If you’re trying to extend the profiler in some way, the task might be easier with this module. Originally designed and written by Jim Roskind.

“Making cProfile / profile as a context manager” is a long-term problem since 2010 [1]，and not yet resolve. The last message on bpo is a cryptic assertion make profile can’t work. Basically what we will do is to add __enter__ and __end__  magic methods into profile’s class Profile, then it can act like a context manager:

But this won’t work, if you try it yourself, you will find that it will pop out a strange assertion error like this:

How could it be? the same trick work with cProfile, but not profile?

 

First, we will need to know how python profiler works. When we want to add a profiler to python program, we will need to use “sys.setprofile” to hook up the profiler and python. Then in each event (call, exception, return, c_call, c_exception, c_return) in python, it will call your profiler to profile. When it call your profiler, it will pass three arguments to the profiler, frame, event and arg. Then your profile will deal with different event, and calculate the time for it.

“profile” module’s class “Profile” is designed to be a pure python profiler, it have a fast dispatcher to dispatch each event to the relative function.

Second, the profiler has a self-maintained frame stack, to record the parent frame for return purpose. This frame stack is build as a 6-item tuple, which is:

The above 6-tuple is the initial tuple for the profile.Profile() instance when created. You can see, the -3 item is ('profile', 0, 'profile') which represent to filename, line number and code object name. The -2 item is a fake frame and -1 is None, indicate that there is no parent, which mean that this is the most top frame that we can return here.

And why we need this, this initial tuple is used to be a sentinel, tell profiler this is the most top of the profiler, and can’t return anymore.

To simulate a deep call like this:

It will build the synthetic frame stack like this:

You must observe the last item, which I have move to start at newline, is the profiler synthetic frame stack. When each call is made, it will record a parent 6-tuple in it.

So in this limitation, if we return at the last frame, which is the tuple that I show it above, I’ll show it again here:

Then this return will trigger a “bad return”, because you can’t return to a None parent, that is an illegal operation.

The Main Problem

We can now return to our main problem: “Why can’t we let profile as a context manager?” here is the code:

Can you tell where get wrong?

Yes, when we get into profiler_helper, the profiler still not hook on python, so the synthetic frame stack it still look like this:

And then we got a return from profiler_helper (we must return back, this function is done), this time, the profiler is hook to python, therefore our dispatcher is work! Then it will try to return,  meet our sentinel and raise a AssertionError that says “Bad Return”.

The Fix

Our fix is very easy, just need to add a simulate call to simulate that we call the profiler_helper:

Then the problem solve, now we can return it correctly!

 

[1]: bpo-9285: Add a profile decorator to profile and cProfile
[2]: bpo-30113: Allow helper functions to wrap sys.setprofile