Eyes are key social features providing a wealth of information about the attention, interest, emotion, and intention of others. Humans are typically very adept at detecting gaze direction, but there is a large decrement in gaze discrimination ability when eye images change from positive to negative polarity. This is thought to show an expert system for gaze perception that applies a contrast-specific heuristic to determine where someone else is looking. Autism spectrum conditions (ASC) are characterized by social deficits including difficulties in face-processing and in the social use of gaze. People with ASC are thought to have less expertise for gaze processing compared to typical controls, though little research has tested this. We investigated gaze direction perception in typical males and females, and males with ASC using facial stimuli with positive or negative polarity of the eyes. Results showed that the ASC group was worse at judging gaze direction with positive stimuli, and showed less of a decrement in performance when eye stimuli changed from positive to negative polarity. The differences in gaze perception for the ASC group were most evident when information from the eyes was more difficult and ambiguous. Typical females performed better at gaze direction detection with positive polarity than typical males, who in turn performed better than males with ASC. This latter finding is consistent with the extreme male brain theory of autism, and with the idea that people with ASC have less gaze expertise.