What your pediatrician may have been referring to is the potential problem of shellfish. Shellfish are a group of fish that are known to be more likely to cause allergic reactions than other foods. Shrimp, lobster, and crab fall into this category. If there is a strong history of allergies in a family, it may be reasonable to avoid shellfish for a while in the child.
The other concern that comes into play with fish is that certain types of fish have been found to have high mercury levels. When the fish is eaten, the mercury is absorbed into the body and adds to the level of mercury for that person. While in adults the small amounts of mercury are likely of little consequence, in children there is concern that these small amounts accumulate over time and can then have a more toxic effect on brain development.
The third thing to consider when giving a young child fish is the bones. Even when a parent attempts to remove all the bones, very often small ones are left, and a young child may have difficulty. He can choke on the bones or have a small piece of bone get caught in the back of the throat.
As you can see, there are a number of factors to consider if you want to give your child fish. It makes sense to avoid bony fish that has higher mercury levels, as well as shellfish. Fish is a wonderful source of protein, however, and provides lots of nutrients and fats that the body does need. A basic whitefish that doesn't have bones and is known not to be significantly contaminated with mercury would be a reasonable choice.