- Antimatter is difficult to get a hold of. You can produce it, but current capabilities would take many millions or even billions of years to produce small amounts. More efficient production may be possible, but even if you got it up to 50%- the maximum possible efficiency (due to baryon conservation or somesuch), it'd still be a massive undertaking to produce the requisite amounts of antimatter. You've basically got to collect the mass-energy of the antimatter, and due to the very same e=mc^2 that makes antimatter such a potent fuel, that's a lot of energy. Imagine a ship that carries 1000 tons of propellant; roughly half of that will be plain old normal matter, the other half antimatter, so 500 tons. Producing that amount of antimatter in a year, assuming the highest physically possible efficiency, would require nearly 3000 terawatts of energy; by comparison, current global power usage is under 20 terawatts. In a decade, around 300 terawatts, in a century, 30. The efficiency of the process will likely be a good deal lower than 50%, so the actual power requirements will be a good deal higher.

In addition, 500 tons of antimatter would go off like hell if it contacted normal matter, so you have to be careful about that (both when you're producing it and when it's onboard the spacecraft). All in all, a massive undertaking.

- If you're utilising the pions produced by the reaction as exhaust (which the beam core design does), you're going to lose a lot of energy to gamma rays, so the efficiency goes down a lot and so will your specific impulse. Gamma rays are really difficult to stop and they're also ionising radiation, which makes them a challenge to deal with.

- An engine that utilised the gamma rays themselves for propulsion would be most efficient- it'd be a photon rocket- but there is no known material that can reflect gamma rays. In addition, total or very near total reflection would be needed- through the thrust power equation, we find that a 1 meganewton engine with an exhaust velocity of 299 792 458 m/s has an output power of about 150 terawatts. Assuming a hemispherical reflector 20 meters across, that absorbs one tenth of one percent of the incident energy, it would reach over 6000 K, which is obviously nonviable.

Now, how valid all of this is, I'm not sure- but I think what I'm getting pretty right is that I've isolated the seriously hot bits of the ship from everything else. The entire ship's surface area can't be used as a radiator, even if it were practical to distribute the heat over its entirety, it wouldn't be the optimal shape (it would probably radiate into itself, dropping efficiency; the best shape for a radiator is a flat plane), and most importantly, not everything can live at say, 1000 K. Cryogenic propellants, human cargo, avionics, and soforth. Another thing that needs to be cooled is the machinery involved with the fusion process- the electromagnets, ignition beams if using an IC design, etc. That's not necessarily a bad thing; with this in mind, designing such a spacecraft may be somewhat more straightforward. However, juggling the waste heat is still physically an issue.

You can waltz around the solar system in just days or weeks at small fractions of a G, for instance. I think the problem is that people get the impression from chemical spacecraft that you've got to accelerate at a considerable pace for a short while, and then coast for most of your voyage, before decelerating for a short while. That sort of flightplan is totally motivated by the characteristics of chemical rockets; low isp, but high thrust. Conversely, a high performance vehicle- say, a fusion powered one- will have high ISP, but probably have a lower rate of acceleration, because it's difficult to build high-isp engines that have a good thrust/weight.

So what you really want to do is accelerate constantly till the halfway point, turn around (or fire an engine in the opposite direction) and decelerate constantly till you get to the destination. This requires the rate of acceleration. Of course, disregarding dV, it now becomes a limitation- a spacecraft with a set acceleration will only be able to traverse a certain distance in a certain amount of time. However, even at fairly low accelerations, these travel times are quite impressive. This calculator, IIRC, calculates maximum velocity and trip time for a given acceleration and distance, assuming continuous acceleration and deceleration. As one can see, an acceleration of only 0.001G is sufficient to reach 10% of c over 10 light years. A spacecraft able to accelerate at 0.001G is far easier to consider than one that accelerates at 10G.

Now, there are many ways that one can play around with it; for instance, in some situations one would indeed stop accelerating at some point and coast the rest of the way before decelerating. It all depends on the dV, acceleration, distance, and top speed that one wants to reach.

At a higher specific impulse, more power is produced per newton of thrust. This is why a Saturn V producing around 40MN at launch puts out roughly 100 gigawatts, whereas a fusion spacecraft with a comparable thrust puts out many terawatts. It's possible, in theory though not necessarily in practice, to trade specific impulse for thrust. For instance, injecting hydrogen (or any other mass) into a fusion spacecraft's exhaust would lower specific impulse and increase thrust; the total mass flow would be higher, and the energy per unit mass would be lower, with overall thrust power remaining the same. Note that this would not be without its inefficiencies and implementation problems, but it's still a possible way of getting more thrust if it is absolutely necessary.

So it's mostly pointless to talk about reflecting most of the waste heat, as there isn't any known material that can reflect X-rays (grazing reflection aside). X-rays do travel through most things, so instead of a mirror one might want to design a 'window', though of course they will still impart some of their energy and heat things up, and various things need to be shielded from X-rays; sensitive equipment, cargo, crew, etc. You'll probably need shielding to absorb radiation and protect various parts of the ship, and that will unfortunately drive mass up. And of course, this adds a new dimension to the problems that pervade high-power spacecraft; not only are you dealing with thermal problems, but you can easily give your passengers many times the lethal dose in a strikingly short amount of time.