No, there is no need to use panels all of the same size (even if huge difference in dimensions will affect precision, as you average the charge on a large panel).
Most probably, instead, there is some error in the input file, relative to the input panel vs. the dielectric reference point.
The test is done (as you can see in the source code) to "determine if normal needs to be flipped to get dielectric boundary condition right".
However, FastCap should have printed some other debugging information, as per this printf statement:
viewprintf(stderr,
"flip_normal: inconclusive test for normal flipping\n");
viewprintf(stderr, " Surface: %s\n", hack_path(surf_name));
viewprintf(stderr, " Translation: (%g %g %g)\n", surf->trans[0],
surf->trans[1], surf->trans[2]);
viewprintf(stderr, " Reference point: (%g %g %g)\n",
ref[0], ref[1], ref[2]);
viewprintf(stderr, " Panel corner: (%g %g %g)\n",
panel->corner[0][0], panel->corner[0][1], panel->corner[0][2]);
viewprintf(stderr, " Normal: (%g %g %g)\n",
normal[0], normal[1], normal[2]);
Please provide also the additional traces, or the smallest offending input file you can generate (please do not C&P a 5k lines input file. You should be able to narrow down the issue to a few panels).
Best Regards,
Enrico