Some sexual abuse of (say) 12-year-olds is committed by pedophiles (or hebephiles). But some is committed by adults whose primary attraction is to other adults. We call them opportunistic offenders. The opportunity was there, and they had enough sexual attraction to engage in sexual activity.
Suppose we also start with the simplest assumption that a person's fundamental sexual attraction, once it becomes clear at the start of puberty, is unchanging. One challenge to this view is that young teens will frequently feel a strong attraction to each other. Society does not find a 12-year-old boy's attraction to his 12-year-old classmate unusual. But if he gets to be 30 and his attraction focuses on 12-year-old girls who are similar to his past classmate, we do judge that to be unusual (at the least). Did his sexual attraction change? I propose that it didn't.
I suggest that his attraction to his classmate at age 12 was opportunistic. In most western societies, we segregate our kids for educational purposes by age, so the girls he rubs shoulders with every day are other 12-year-olds. They also share much of their life experience and situation. Realistically, he doesn't have much chance with an 18-year-old girl anyway and focuses his attention where he does have an opportunity, perhaps. His fundamental attraction can be seen in his choice of which models and movie stars he finds to be hottest. They will tend to be adults, not 12-year-old girls.
All the same considerations apply to girls. However, by the time they are 14 (typically) some do have realistic opportunities with older men. Men find them genuinely attractive. And to the girl (as to older females as well), the 25-year-old man is much more attractive than her 14-year-old boy classmate. Society is horrified when a 25-year-old man has a relationship with a 14-year-old girl. But the 14-year-old girl is often an enthusiastic participant (though it still may end badly a great deal of the time).
We can also find opportunistic attraction when we come to people (especially men) in their 40s and above. They still have their strongest attraction to women in their 20s. If men between the ages of 20 and 70 find women between 20 and 30 to be most attractive, that's a lot more men than women, and most men will have to make do with older women. They also will share more of their life experience and connect better on levels that are not gut-level sexual attraction.
Women also may feel the strongest gut-level physical attraction to men in their 20s, but women seem wired to put a higher value on material success than physical appearance. The men in their 40s and 50s who do marry women in their 20s tend to be successful alpha males. That is why the 20-something women chose them.
So we can retain the view that fundamental sexual attraction is set by early puberty and the age distribution does not change as a person grows older. When it comes to forming actual relationships, people will tend to match up with those their own age due to shared experience and life outlook. But another key reason is simply opportunity. Teens in particular make do with each other because they are together so much. Older people also do the best they can simply based on who is available to them.