It would be awesome to be able to change the order of categories so that when one adds a new category one can place it next to the related one. Currently, they appear in the order they were added and there is no way one can rearrange the order.
Also, it should be possible to delete a category without also deleting all of the posts that were tagged in the category. Currently, if one wants to delete a category that also means that all of the posts in this category will be deleted, this really doesn't make sense.