usually you will face the following error: Underlying DBMS error[ORA-00001: unique constraint] while compressing the ArcSDE Geodatabase. to know more information about the constraint index that is causing the issue, check the ArcSDE log file located in your $SDEHOME/etc directory.

once you identify the constraint number and feature class or table, use the ArcSDE gdbrepair utility to fix this issue.

command syntax:

sdegdbrepair -o repair_tables -d <{ORACLE10G|ORACLE11G|SQLSERVER|DB2|DB2ZOS|INFORMIX|POSTGRESQL}>
[-r versioned table name | file=<table_list>] [-V {ALL | NONE | <version_name>} | STATE:<state_id>]
[-H <home_directory>] [-u <ArcSDE_admin_user>] [-p <ArcSDE_admin_password>]
[-D <database_name>] [-s <server_name>] [-i <master_geodatabase_service>:<user_schema>] [-N] [-q]

Remarks:

its better to use the flag -V ALL

also, STATE:state id is the constraint number shown in the error.

ESRI Link References:

http://support.esri.com/fr/knowledgebase/techarticles/detail/35298

http://help.arcgis.com/en/geodatabase/10.0/admin_cmds/support_files/serveradmin/sdegdbrepair.htm