different single post page for different category

I have posts that are in more than one category. I have different category templates for each of the categories. This works for the category pages. I also have a statement that says that if the post is in a specific category, use a specific single post page (single1, single2, etc.). The problem is – when I click on the permalink on the category page for a post that is in more than one category, it doesn’t display the correct single post template. For example, a post in Awards category and Agriculture category is supposed to display using a different single post template, depending on the category. How do I make sure that the category that calls the post also calls the correct single post template? Thanks.