Display Sub-Categories On Single.php

I have a series of categories as follows:

Books
Music
Kids
Video

Each of those have sub-categories… such as under “Books” there is…

Arts
Biography
Books on Tape/CD
Business & Finance

I’m using the following code to display a list of sub-categories of the main category on the sidebar.

So, in other words, when you go to “Books” the sidebar shows the sub-categories listed above. That is done with this code…

<?php
// in category template, get child cats and posts
$cat = get_query_var('cat');
$cat_args=array(
  'child_of' => $cat,
  'hide_empty' => 0,
  'orderby' => name,
  'style' => none,
  'title_li' => '',
   );
$categories=wp_list_categories($cat_args);
?>

When you click on a specific book, however, and are taken to the “single” page for that entry, it displays ALL of the categories on the sidebar.

I want to display only the sub-categories of the main category of which you are under.

So, if you click on a single book, it shows the sub-categories of the “Books” category, and so on.

Ideas?