To what extent is the size of the blood-oxygen-level-dependent (BOLD) response influenced by factors other than neural activity? In a re-analysis of three neuroimaging datasets (male and female human participants), we find large systematic inhomogeneities in the BOLD response magnitude in primary visual cortex (V1): stimulus-evoked BOLD responses, expressed in units of percent signal change, are up to 50% larger along the representation of the horizontal meridian than the vertical meridian. To assess whether this surprising effect can be interpreted as differences in local neural activity, we quantified several factors that potentially contribute to the size of the BOLD response. We find relationships between BOLD response magnitude and cortical thickness, curvature, depth and macrovasculature. These relationships are consistently found across subjects and datasets and suggest that variation in BOLD response magnitudes across cortical locations reflects, in part, differences in anatomy and vascularization....