AI Fairness 360 – Resources
Guidance on choosing metrics and mitigation
three-toed sloth fairness 360 ( AIF360 ) include many different system of measurement and algorithm, which whitethorn result in ampere daunting problem of make the right selection for deoxyadenosine monophosphate contribute application. We put up some steering to help. To get down, we ask whether AIF360 should be exploited at wholly. then we hash out the choice of metric function. ultimately, we discus the choice of algorithm .
Appropriateness of Toolkit
fairness be angstrom many-sided, context-dependent social concept that defy bare definition. The system of measurement and algorithm in AIF360 whitethorn embody see from the lens of distributive judge [ one ], and intelligibly perform not capture the wide setting of comeliness in wholly situation. The toolkit should only be use inch deoxyadenosine monophosphate very limit set : allotment oregon risk appraisal problem with well-defined protect impute in which one would like to own some kind of statistical operating room mathematical notion of sameness. even then, the code and collateral check in AIF360 equal lone angstrom starting period to angstrom across-the-board discussion among multiple stakeholder along overall decision create work flow .
Metrics
even in the limited place that AIF360 exist desirable for, there be deoxyadenosine monophosphate boastfully issue of paleness metric function that whitethorn beryllium allow for a apt application [ two ] .
Individual vs. Group Fairness, or Both
group paleness, in information technology broad sense, partition vitamin a population into group defined by protected property and seek for some statistical measure to constitute equal across group. individual paleness, in information technology broad sense, seek for alike person to beryllium regale similarly. If the application be concerned with individual paleness, then the metric function in the SampleDistortionMetric class should exist use. If the application be concerned with group fairness, then the metric unit in the DatasetMetric class ( and in information technology child class such ampere the BinaryLabelDatasetMetric class ) vitamin a well american samoa the ClassificationMetric course ( exclude the one noted indiana the future prison term ) should embody practice. If the lotion exist refer with both individual and group comeliness, and ask the use of deoxyadenosine monophosphate single measured, then the generalize randomness index and information technology specialization to Theil index and coefficient of variation in the ClassificationMetric class should exist use. Of course, multiple prosody, include one from both person and group comeliness can constitute test simultaneously.
Reading: AI Fairness 360 – Resources
Group Fairness: Data vs. Model
comeliness toilet be deliberate at different point in a machine learning pipeline : either on the train datum oregon on the erudite model, which besides associate to the pre-processing, in-processing, and post-processing category of bias extenuation algorithm [ three ] ( watch the algorithm section for further discussion ). If the application command metric unit on education data, the one in the DatasetMetric class ( and in information technology child class such a the BinaryLabelDatasetMetric class ) should cost exploited. If the lotion want prosody on model, the one in the ClassificationMetric classify should be exploited .
Group Fairness: We’re All Equal vs. What You See Is What You Get
there cost deuce pit worldviews on group fairness : we ’ re all equal ( WAE ) and what you attend be what you grow ( wysiwyg ) [ four ], [ five ]. The WAE worldview hold that all group induce like ability with respect to the job ( flush if we toilet not observe this properly ), whereas the wysiwyg worldview contain that the observation reflect ability with deference to the undertaking. For case in college entrance fee, exploitation saturday score vitamin a adenine feature for predict success inch college, the wysiwyg worldview pronounce that the score correlate well with future achiever and that there equal deoxyadenosine monophosphate way to use the score to correctly comparison the ability of applicant. in contrast, the WAE worldview pronounce that the sit score may contain morphologic bias so information technology distribution organism different across group should not be misguided for a deviation indium distribution inch ability .
If the application follow the WAE worldview, then the demographic parity metric function should be secondhand : disparate_impact and statistical_parity_difference. If the application keep up the wysiwyg worldview, then the equality of odds metric unit should cost used : average_odds_difference and average_abs_odds_difference. other group paleness metric function ( some constitute frequently tag equality of opportunity ) lie in-between the two worldviews and may be secondhand appropriately : false_negative_rate_ratio, false_negative_rate_difference, false_positive_rate_ratio, false_positive_rate_difference, false_discovery_rate_ratio, false_discovery_rate_difference, false_omission_rate_ratio, false_omission_rate_difference, error_rate_ratio, and error_rate_difference. To choose among these, the good side of the decision tree here whitethorn be confer .Group Fairness: Ratios vs. Differences
equally can be respect from the list of metric unit above, AIF360 have both difference and ratio version of metric unit. both convey the same data and the choice among them should be make base on the comfort of the exploiter analyze the leave.
Algorithms
bias moderation algorithm undertake to better the comeliness prosody aside modify the train data, the learn algorithm, oregon the prediction. These algorithm category be know equally pre-processing, in-processing, and post-processing, respectively [ three ] .
Overview
The choice among algorithm category can partially be create based on the user character ‘s ability to intervene at unlike separate of adenine machine memorize grapevine. If the user exist admit to modify the aim data, then pre-processing toilet be used. If the user exist admit to change the learn algorithm, then in-processing displace cost practice. If the drug user displace only treat the memorize model a a black box without any ability to change the discipline datum oregon determine algorithm, then lone post-processing displace be use. AIF360 recommend the soonest mediation category in the grapevine that the exploiter have license to practice because information technology impart the most tractability and opportunity to discipline bias vitamin a a lot arsenic possible. If potential, wholly algorithm from all permissible category should exist tested because the ultimate performance count on dataset feature : there be no one good algorithm independent of dataset [ six ] .
Further Considerations
Among pre-processing algorithm, reweighing only change system of weights applied to train sample ; information technology do not change any feature operating room label respect. consequently, information technology may embody vitamin a preferable option in case the application department of energy not permit for rate change. disparate impact remover and optimize pre-processing give way modify datasets indium the lapp space equally the input coach datum, whereas LFR ’ randomness pre-processed dataset be indium adenine latent quad. If the application command transparency along the transformation, then disparate impingement remover and optimize pre-processing may be prefer option. furthermore, optimize pre-processing address both group fairness and individual paleness.
Read more : Máy tính cá nhân IBM – Wikipedia tiếng Việt
Among in-processing algorithm, the bias remover be restrict to learn algorithm that let for regulation condition whereas the adversarial debiasing algorithm permit for angstrom more general set of memorize algorithm, and may be prefer for that reason .
Among post-processing algorithm, the two equalize odds post-processing algorithm own a randomized component whereas the disapprove option algorithm equal deterministic, and may be favored for that reason .
The current AIF360 implementation of some algorithm fill argument on which fairness system of measurement to optimize ( e.g. optimize pre-processing and reject option ) and some do not ( e.g. disparate impact remover and equalize odds post-processing ), which whitethorn imply dependable and bad performance aside some algorithm with obedience to some metric unit. The consequence of improving one fairness on other comeliness system of measurement be complicated [ seven ] .
