It depends. Is this a first occurrence, or a reoccurring problem?
If it's the first time, I'm going to attribute it to spring fever. The scenario also says that I took the day off early as well. In that case, I may ask him to give me a hand before letting him return to the pool. I'm going to read him the riot act, though--truancy is illegal, and it better not happen again no matter how nice the day. And by today's standards, swimming alone is also a "safety issue"--though I swam alone from age 12,and still usually do if I swim at all because there isn't anyone else around.
If it's a reoccurring problem, I'm going to dry him off and even him out.