Ethical AI Frameworks, Guidelines, Toolkits

AI Frameworks,
Guidelines,
Toolkits

Frameworks, Guidelines, Toolkits

ACT-IAC Emerging Technology COI, White Paper: Ethical Application of AI Framework, (2020)

ACT-IAC Emerging Technology COI, AI Playbook for the US Federal Government, (2020)

Ada Lovelace Institute, AI Now Institute and Open Government Partnership. Algorithmic Accountability for the Public Sector, (2021)

AI4People (Atomium - European Institute for Science, Media and Democracy), Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations

AI4People (Atomium - European Institute for Science, Media and Democracy), On Good AI Governance 14 Priority Actions, a S.M.A.R.T. Model of Governance, and a Regulatory Toolbox

AI4People (Atomium - European Institute for Science, Media and Democracy), 7 AI Global Frameworks

AI Global: Responsible‌ ‌Design‌ ‌Assistant‌

AI Now Institute, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability

AI Now Institute, Algorithmic Accountability Policy Toolkit

AISP, A Toolkit for Centering Racial Equity Throughout Data Integration

Alan F. Winfield ; Katina Michael ; Jeremy Pitt ; Vanessa Evers: Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems, Published in: Proceedings of the IEEE ( Volume: 107 , Issue: 3)

Alan Turing Institute and the Information Commissioner’s Office (ICO), Project ExplAIn

Algofairness, BlackBox Auditing Tool

AlgoritmWatch, Ethics and algorithmic processes for decision making and decision support

Allied Media: People’s Guide to AI

All Tech is Human: Guide to Responsible Tech: How to Get Involved & Build a Better Tech Future

Amazon - AWS Conditional Demographic Disparity (CDD)

Amazon - AWS Responsible AI training modules

Amazon - AWS AI Service Cards

Amazon - AWS SageMaker Model Cards

Ambacher, B. u. a. , Trustworthy Repositories Audit & Certification: Criteria and Checklist (TRAC), CRL Center for Research Libraries, Chicago, (2007)

Arogyaswamy, B. Big tech and societal sustainability: an ethical framework. AI & Soc (2020)

Australian Human Rights Commission: Using artificial intelligenceto make decisions:Addressing the problemof algorithmic bias

ATARC: Artificial Intelligence and Data Analytics (AIDA) Guidebook

B. d’Alessandro, C. O’Neil, T. LaGatta, Conscientious classification: A data scientist’s guide todiscrimination-aware classification, Big data 5 (2) (2017)

Barcelona City Council Open Digitisation Plan

Barocas, Solon and Hood, Sophie and Ziewitz, Malte, Governing Algorithms: A Provocation Piece, (2013)

BASIC - A Toolkit and Ethical guidelines for Applying Behavioural Insights in Public Policy, OECD, (2018)

Ben Shneiderman (2020) Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy, International Journal of Human–Computer Interaction, 36:6

Bethge Lab: AI Foolbox

Berkley Haas: Mitigating Bias in Artificial Intelligence Playbook

Brown University, A Framework for Making Ethical Decisions

BSA: Confronting Bias: BSA’s Framework to Build Trust in AI

Brundage, Miles et al. “Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims.” ArXiv abs/2004.07213 (2020)

Canada Algorithmic Impact Assessment

Centre for Applied Data Ethics, Ethical Self-assessment Tool

Center for Democracy & Technology (CDT), DD Tool

Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. When the Algorithm Itself Is a Racist: Diagnosing Ethical Harm in the Basic Components of Software, Int’l. J. Comm. 10: (2016)

Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” In Data and Discrimination: Converting Critical Concerns into Productive: A preconference at the 64th Annual Meeting of the International Communication Association. Seattle, WA, (2014)

CIGREF, Digital Ethics

Council of Europe: Human Rights, Democracy, and the Rule of Law Assurance Framework for AI Systems

Credo AI Lens: comprehensive assessment framework for AI systems. Lens standardizes model and data assessment, and acts as a central gateway to assessments created in the open source community

Dafoe, Allan. "AI Governance: A Research Agenda", 2018

Databaseline, ML Cards for D/MLOps Governance

Data Cards Playbook

DataEthics, White Paper on Data Ethics in Public Procurement of AI-based Services and Solutions, 2020

DataEthics, Impact Assessment, 2021

Data for Children Collaborative with UNICEF, Ethical Assessment

Data Responsibly Toolkit

Data Society, Algorithmic Impact Assessment for the Public Interest

David Freeman Engstrom, Stanford University Daniel E. Ho, Stanford University, Catherine M. Sharkey, New York University, Mariano-Florentino Cuéllar. "Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies"

Dawson D and Schleiger E*, Horton J, McLaughlin J, Robinson C∞, Quezada G, Scowcroft J,and Hajkowicz S†(2019) Artificial Intelligence: Australia’s Ethics Framework. Data61 CSIRO, Australia.

Deepcheck, Test Suites for Validating ML Models & Data

Dent, Kyle and Dumond, Richelle and Kuniavsky, Mike, A Framework for Systematically Applying Humanistic Ethics when Using AI as a Design Material (July 1, 2019). Temes de Disseny 35

Design Justice for Action

Diakopoulos, N., “Algorithmic Accountability Reporting: On the Investigation of Black Boxes”, (2014)

Digital Catapult: Ethics Framework

doteveryone, Consequence Scanning Kit

D. Peters, K. Vold, D. Robinson and R. A. Calvo, "Responsible AI—Two Frameworks for Ethical Design Practice," in IEEE Transactions on Technology and Society, vol. 1, no. 1, pp. 34-47, March 2020

DQ Institute: DQ Framework Global Standards on Digital Literacy and Skills

Driven Data, Deon ethics checklist

Ethics Canvas

Expert Group Data Ethics, Ethical Codex for Data- Based Value Creation, (2019)

European Commission High Level Expert Group on AI, Ethics Guidelines for Trustworthy AI, 2019

European Commission High-Level Expert Group on AI, Trustworthy AI Assessment List

European Commission High-Level Expert Group on AI, Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment

European Commission High-Level Expert Group on AI, Towards a European strategy on business-to-governmentdata sharing for the public interest

European Commission, White Paper on Artificial Intelligence -A European approach to excellence and trust

European Commission & ADAPT, EnTIRE (Mapping Normative Frameworks for EThics and Integrity of REsearch)

European Council Committee of Ministers, Recommendation CM/Rec(2020)1 of the Committee of Ministers to member States
on the human rights impacts of algorithmic systems, 2020

European Parliament, A governance framework for algorithmic accountability and transparency, (2019)

EU Privacy Seals Project

Facets, Analyzing machine learning datasets by visualization

FactSheets: Increasing Trust in AI Services through Supplier’s Declarations of Conformity

FairLearn, Open source Python toolkit

FairSight, Fair DecisionMaking Tool

FBPML Wiki, Best Practices in ML

F. Kamiran, T. Calders, Data preprocessing techniques for classification without discrimination, Knowledge and Information Systems 33 (1) (2012)

Floridi, L. (2020). Ethical Foresight Analysis: What it is and Why it is Needed. Minds and Machines

Floridi, L. (2020). How to Design AI for Social Good: Seven Essential Factors

Fujitsu, AI Ethics Impact Assessment & Toolkit

GAO (Government Acountability Office) AI Accountability Framework for Federal Agencies and Other Entities

GitHub, CodeSearchNet

Global AI Standards Repository

Google, Playing with AI Fairness: What-if Tool

Google, Dataset Search Beta (aiEthicist.org does not suggest that the datasets are free of bias, but only provides link to this Google tool)

Google, Explainable AI

Google, Perspectives on Issues in AI Governance

Google, ML-fairness-gym: A Tool for Exploring Long-Term Impacts of Machine Learning Systems

Google, Know Your Data

GovEx, the City and County of San Francisco, Harvard DataSmart, and Data Community DC, Ethics and Algorithms Toolkit

GovLab, Re-imagining Governance in PracticeBenchmarking British Columbia’s Citizen Engagement Efforts, 2013

GovLab, The Open Policy Making Playbook

GovLab, Mapping and Comparing Responsible Data Approaches, 2016

Government of the Netherlands, Fundamental Rights and Algorithms Impact Assessment (FRAIA)

Hallensleben, Sebastian and Hustedt, Carla. From Principles to Practice: An interdisciplinary framework to operationalise AI ethics, Bertelsmann Stiftung 2020

Harvard University - Berkman Klein Center for Internet and Society, Ethics and Governance of AI

HUB4NGI, Responsible AI – Key Themes, Concerns & Recommendations for European Research and Innovation Summary of Consultation with Multidisciplinary Experts (2018)

Humanitarian Data Science and Ethics Group (DSEG), A Framework for the Ethical Use of Advanced Data Science Methods in the Humanitarian Sector

Information Accountability Foundation (IAF), Comprehensive Publications About Information Accountability

IAPP, Building Ethics into Privacy Frameworks for Big Data and AI

IBM, AI Fairness 360 Open Source Toolkit

IBM, AI Explainability 360 Toolkit

IBM, AI FactSheets 360 Toolkit

IBM, IBM Watson OpenScale

IBM, Everyday Ethics for Artificial Intelligence

IBM, Advancing AI ethics beyond complianceFrom principles to practice

IBM, Adversarial Robustness Toolbox

IBM, inFairness

ICAEW: New technologies, ethics and accountability

ICO, Guidance on the AI auditing framework

ICO, Guidance on AI and data protection

IDEO, AI Ethics Cards

IEEE, Ethically Aligned Design, Version 1 (EADv1)

IEEE, Global Initiative on Ethics of Autonomous and Intelligent Systems. Ethically Aligned Design: A Vision for Prioritizing Human
Well-being with Autonomous and Intelligent Systems, Version 2 - Request for Input. IEEE, 2017.

IEEE, CertifAIEdTM certification program

IFTF & Omidyar Network, Ethical OS Framework

Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell,Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona,Spain .ACM, New York

Institute for Ethical AI & ML, AI-RFX Procurement Framework

Institute and Faculty of Actuaries (IFoA) and the Royal Statistical Society (RSS), A Guide for Ethical Data Science

Institute for the Future of Work, Policy Briefing on algorithmic impact assessments

interpretML

ISO/IEC JTC 1/SC Standardization on Artificial Intelligence

Japanese Society for AI (JSAI), Ethical Guidelines

Jessica Morley, Luciano Floridi, Libby Kinsey, Anat Elhalal. From What to How: An Overview of AI Ethics Tools, Methods and Research to Translate Principles into Practices and Supporting Material

Jobin, A., Ienca, M. & Vayena, E. ,The Global Landscape of AI Ethics Guidelines, Nat Mach Intell 1, (2019)

Judy Goldsmith, Emanuelle Burton, Why Teaching Ethics to AI Practitioners Is Important, AAAI Conference on Artificial IntelligenceThirty-First AAAI Conference on Artificial Intelligence, 2017

Kaminski, Margot E. and Malgieri, Gianclaudio, Algorithmic Impact Assessments under the GDPR: Producing Multi-layered Explanations. U of Colorado Law Legal Studies Research Paper No. 19-28. (2019)

Kat Zhou, Design Ethically Toolkit

Kathy Baxter (Salesforce Research), How to Build Ethics into AI — Part I Research-based recommendations to keep humanity in AI

Kathy Baxter (Salesforce Research): Building Ethics into AI: Lessons Learned from Pioneers in the Trenches

Kiritchenko, Svetlana and Mohammad, Saif M., Equity Evaluation Corpus (EEC)

Koshiyama, Adriano and Kazim, Emre and Treleaven, Philip and Rai, Pete and Szpruch, Lukasz and Pavey, Giles and Ahamat, Ghazi and Leutner, Franziska and Goebel, Randy and Knight, Andrew and Adams, Janet and Hitrova, Christina and Barnett, Jeremy and Nachev, Parashkev and Barber, David and Chamorro-Premuzic, Tomas and Klemmer, Konstantin and Gregorovic, Miro and Khan, Shakeel and Lomas, Elizabeth, Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms (February 4, 2021)

Kriebitz, A., & Lutge, C. Artificial Intelligence and Human Rights: A Business Ethical Assessment. Business and Human Rights Journal, 5(1), 84–104. Cambridge University Press (2020)

Leidner, Jochen L. and Vassilis Plachouras. Ethical by Design: Ethics Best Practices for Natural Language Processing

Leighton Andrews, Bilel Benbouzid, Jeremy Brice, Lee A. Bygrave, David Demortain, Alex Griffiths, Martin Lodge, Andrea Mennicken, Karen Yeung. Algorithmic Regulation. The London School of Economics and Political Science, (2017)

Leslie, D. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector, The Alan Turing Institute, (2019)

LinkedIn Fairness Toolkit (LiFT)

Linux Foundation, Apache NiFi ‹› AI Fairness 360 (AIF360) Integration – Trusted AI Architecture Development Report 1

Linux Foundation, AI Explainability 360

Lütge, Christoph. AI Ethics and Governance “Building a Connected, Intelligent and Ethical World”, 2020

Madaio, Michael A. and Jennifer Wortman Vaughan. “Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI.” (2020).

Markkula Center for Applied Ethics, A Framework for Ethical Decision Making

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). Association for Computing Machinery, New York

MetaAI, Fairness Flow

Microsoft, AI Fairness Checklist

Microsoft, Judgment Call

Microsoft, Error Analysis

Microsoft, Responsible AI Toolbox

Microsoft, Human-AI eXperience (HAX) Toolkit

Ministry of Economy, Trade and Industry (METI) of Japaan: Contract Guidelines on Utilization of AI and Data

Moral Framework for Understanding of Fair ML through Economic Models of Equality of Opportunity

NetHope Solutions Center, Artificial Intelligence (AI) Suitability Toolkit for Nonprofits

NIST, Artificial Intelligence and User Trust (NISTIR 8332)

NIST, A Plan for Federal Engagement in Developing Technical Standards and Related Tools

NIST, AI Risk Management Framework (AI RMF)

NIST, AI Risk Management Playbook - draft

OECD, Framework for the Classification of AI Systems

Office of the Privacy Commissioner for Personal Data, Ethical Accountability Framework for Hong Kong

Open Data Institute, The Data Ethics Canvas

Open Ethics Canvas

Open Robo Ethics, AI Ethics Assessment Toolkit

Oracle, Skater

Oxford Handbook of Ethics of AI: Online Supplement

PARC, AI Ethics Review

Partnership on AI, About ML

Platform for the Information Society, Artificial Intelligence Impact Assessment

PrepareCenter.org: Data Playbook

Princeton, REVISE: REvealing VIsual biaSEs

ProPublica, Data Store

pymetrics, audit AI

PWC: Responsible AI Toolkit

R. Benjamins, A. Barbado, D. Sierra, Responsible AI by design

Responsible Research and Innovation, Self-Reflection Tool

Responsible Innovation Compass, Self-Check

Rolls Royce, AIetheia Framework

Ruf, B., Boutharouite, C., & Detyniecki, M. Getting Fairness Right: Towards a Toolbox for Practitioners.

Sabou, Marta & Bontcheva, Kalina & Derczynski, Leon & Scharl, Arno. (2014). Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC'14).

Salesforce, AI Economist, Simulation Card Foundation Economic Simulation Framework

Santa Clara University, Markkula Center, An Ethical Toolkit for Engineering/Design Practice

Sasha Costanza-Chock, Maya Wagoner, Berhan Taye, Caroline Rivas, Chris Schweidler, Georgia Bullen, & the T4SJ Project, 2018. #MoreThanCode: Practitioners reimagine the landscape of technology for justice and equity. Research Action Design & Open Technology Institute.

Scikit-Fairness Tool

SciPy 2021 Tutorial Fairness in AI systems: From social context to practice using Fairlearn

Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020).

SHAP (SHapley Additive exPlanations)

SHERPA (Shaping the ethical dimensions of smart information systems–a European perspective): Guidelines for the Ethical Development of AI and Big Data Systems: An Ethics by Design approach

Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Model Artificial Intelligence Governance Framework, 2nd Edition

Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Implementation and Self-Assessment Guide for Organizations

Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: AIverify - AI Governance Testing Framework & Toolkit

Smart Dubai, AI System Ethics Self-Assessment Tool

Stanford Center on Philanthropy and Civil Society, Digital Impact Toolkit

Stark, Jeannette and Nicholas Diakopoulos. “Algorithm Tips : A Resource for Algorithmic Accountability in Government.” (2017)

TensorFlow, Fairness Indicators

TensorFlow, Model Analyzer

TensorFlow, Responsible AI - Model Remediation

Thilo Hagendorff, The Ethics of AI Ethics -- An Evaluation of Guidelines

Tilburg University, Handbook on Non-discrimination by Design

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford, "Datasheets for Datasets," (2018)

Timnit Gebru, Emily Denton, FATE/CV Tutorial (Fairness, Accountability, Transparency, Ethics / Computer Vision)

UC Berkeley Center for Long-Term Cybersecurity (CLTC), Decision Points AI Governance

UK Government: UK Data Ethics Framework

UK Government: Ethics, Transparency and Accountability Framework for Automated Decision-Making

UK Government: Artificial Intelligence and Public Standards Report

UK Government: Algorithms: How they can reduce competition and harm consumers

UK Government: Algorithmic Transparency Standard

UK Government: Al Assurance Guide & Roadmap

UnBias Fairness Toolkit

UNDP, UN Global Pulse, ‘A Guide to Data Innovation for Development: From Idea to Proof of Concept,’ 2016

University of Chicago Center for Data Science and Public Policy, Aequitas Bias & Fairness Audit

University of Washington, LIME

Urban Institute: Spatial Equity Data Tool

US Health & Human Services - Trustworthy AI Playbook

Utrecht University, Data Ethics Decision Aid (DEDA)

Value Sensitive Design and Information Systems

Vidgen, R., Hindle, G., and Randolph, I., (2020). Exploring the ethical implications of business analytics with a business ethics canvas. European Journal of Operational Research, 281(3)

Vollmer, Sebastian & Mateen, Bilal & Bohner, Gergo & Király, Franz & Ghani, Rayid & Jonsson, Pall & Cumbers, Sarah & Jonas, Adrian & McAllister, Katherine & Myles, Puja & Granger, David & Birse, Mark & Branson, Richard & Moons, Karel & Collins, Gary & Ioannidis, John & Holmes, Chris & Hemingway, Harry. (2018). Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness.

World Economic Forum (WEF) with Centre for the Fourth Industrial Revolution Fellows from Accenture, BBVA, IBM, Suntory Holdings, Australian Institute of Company Directors, Best Practice AI, Latham & Watkins, and Splunk, with contributions from AI4All, AI Board Toolkit

World Economic Forum (WEF) Unlocking Public Sector AI Toolkit: AI Procurement in a Box

World Economic Forum (WEF) AI Procurement in a Box: AI Government Procurement Guidelines

World Economic Forum (WEF) AI Procurement in a Box: AI Procurement in a Box: Workbook

World Economic Forum (WEF) Responsible Use of Technology

World Economic Forum (WEF) AI Governance - A Holistic Approach to Implement Ethics into AI

World Economic Forum (WEF) C-Suite Toolkit

Zuiderwijk, Anneke and Chen, Yu-Che and Salem, Fadi (2021), Implications of the Use of Artificial Intelligence in Public Governance: A Systematic Literature Review and a Research Agenda, Government Information Quarterly

10 Simple Rules for Responsible Big Data Research

510, F.A.C.T Score for Responsible AI