Theoretical models and experimental data have suggested that inbreeding may act to purge deleterious alleles of large effect from populations. Deleterious alleles of small effect, however, may become fixed by inbreeding. Plants have evolved numerous adaptations to maximise heterozygosity in their progeny and limit the effect of such deleterious alleles. Isotoma petraea F. Muell. represents a good model to examine the evolution of one such adaptation, complex hybridity, which incorporates genome coalescence and seed abortion.

Complex hybridity is a genetic system in which true breeding translocation heterozygosity is maintained by autogamy and a balanced lethal system. Isotoma petraea (2n=14) is normally outbreeding with seven bivalents (7II) at meiosis. The Pigeon Rock population is home to both 7II lineages and translocation heterozygotes (with a ring of six plus four bivalents) characterised by varying levels of seed abortion. Complex hybridity was previously proposed to have evolved in I. petraea at Pigeon Rock in response to the combination of extreme inbreeding and accumulation of deleterious mutations. It is suggested to have then migrated to populations southwest of Pigeon Rock to produce hybrids incorporating additional chromosomes into the rings.

RAPD data were collected for individuals representing structural homozygote populations, the Pigeon Rock lineages, and a range of larger-ringed complex hybrid populations. The occurrence of many complex hybrid specific RAPD fragments indicated that a single origin of the genetic system was likely. Cladistic analysis including the closely related I. axillaris as an outgroup fully supported the evolutionary model outlined above. The evolution of complex hybridity in I. petraea clearly demonstrates the advantages of coalescing the genome into fewer and larger units in response to accumulated genetic load. Restricting recombination may elevate the proportion of heterozygous progeny so that ultimately, if the whole genome segregates as a biallelic supergenic locus, up to half of the seed produced on selfing may be fully heterozygous for their deleterious recessives.

Heterogamy, where certain chromosomal components are transmitted only via the egg or via the pollen, may be regarded as another type of genomic coalescence. It has been proposed to operate, in association with monad pollen development, in the tribe Styphelieae of the Epacridaceae. AFLPs generated from pollen (haploid) and sporophytic (diploid) DNA were compared for a number of Styphelieae species, for some putatively non-heterogamic Epacrideae species, and for I. petraea expressing a range of genetic systems. No evidence was found for heterogamy in the Epacrideae specimens. The absence of numerous diploid-derived AFLP markers from the haploid trace suggested the operation of a heterogamous mechanism in Leucopogon conostephioides (Styphelieae) and in an I. petraea complex hybrid sample.

The fixed heterozygosity associated with complex hybridity has implications for the distribution of genetic diversity within and between populations. This was studied using dominant markers (RAPDs and AFLPs), and the results compared to allozyme data. Similar levels of population differentiation (G_ST >85%) were obtained for all three marker systems when calculated using Shannon's index to measure phenotypic variation. However, Nei's G_ST was about 40% when calculated based on allozyme allele frequency data. This illustrates the way in which fixed heterozygosity may lead to an overestimate of diversity within populations when assessed using codominant markers, or methods which estimate allele frequencies for dominant markers based on the assumption of Hardy-Weinberg equilibrium. The level of diversity within populations was related to the breeding system, with inbreeding populations (both 7II and complex hybrid) showing lower diversity than outbreeding populations for both RAPD and AFLP markers.

The results obtained for I. petraea and a range of other Australian native plant species indicate that they do not conform to classical genetic expectations. The distribution of diversity in these species is discussed with respect to recent models for inbreeding depression and is seen primarily as reflecting the ways in which they manage their genetic load.