Prioritizing Transparency to Prevent Epistemic Injustice in Algorithm-Driven Healthcare

As Artificial Intelligence (AI) usage rises in healthcare, it is vital to analyze how AI harms patients and how to mitigate these harms (European Commission, 2021). In Mitigating Racial Bias in Machine Learning, Kostick-Quenet et al. critique existing regulatory approaches for ineffectively addressing injustices perpetrated by AI algorithms (2022, p. 96). Kostick-Quenet et al. underscore racial bias affecting datasets in a left ventricular-assist device (LVAD) outcome prediction Machine Learning algorithm to act as an example of how AI can perpetuate injustices (2022, p. 93-95).

Epistemic injustice, as defined by Miranda Fricker, occurs when individuals are wronged in their capacity as knowers, either through having their credibility wrongfully diminished (testimonial injustice) or through being denied the resources to describe their experiences (hermeneutical) (2007, p. 1). The injustices described by Kostick-Quenet et al. are simultaneously testimonial and hermeneutical. Patient concerns about the algorithm’s predictions may be dismissed due to the algorithms’ perceived authority (Shekar et al., 2024) or ‘race-neutral’ status (Kostick-Quenet et al., 2022, p.97). Its hermeneutical aspect stems from the fact that patients are denied access to the reasoning behind their predictions and cannot conceptually engage with the care they are receiving (Kostick-Quenet et al., 2022, p. 97-98).

Building on their framework, this application paper identifies a contradiction between Intellectual Property (IP) protections for proprietary algorithms and HIPAA’s expressed goal of “individuals' privacy rights to understand and control how their health information is used.” (U.S. Department of Health and Human Services, 2022). Through a case study on Control-IQ (CIQ), an insulin dosing software for T1D, I demonstrate how IP protections undermine HIPAA’s goal by inhibiting patient access to a system that uses their health information with constrained input. Without interpretable algorithmic output values, patients are unable to effectively control how their health information impacts dosing decisions. These injustices mirror the injustices revealed by the LVAD case study as they are simultaneously testimonial and hermeneutical in nature. To resolve this contradiction, I argue that developers of algorithms used in dosing guidelines should be required to offer disclosure of algorithms under NDA agreements, similar to Employee NDA agreements. This solution protects developer trade interests while offering patients the opportunity to supervise their care as outlined by HIPAA.

Kostick-Quenet et al. begin their piece with a short paragraph describing the prognostic ML algorithm that calculates personalized patient outcomes after receiving an LVAD (2022, p. 93). The expressed goals of this ML algorithm are “to help clinicians identify suitable candidates for LVAD and help inform patients about treatment decisions.” (2022 p. 93). The intention for patient disclosure is vital as it establishes that the injustices that result from racial bias within the algorithm will result in epistemic injustice towards the patients. The final two sentences of this paragraph establish that their case study may be applied to other cases and “highlight gaps in existing regulations to guide developers and mitigate algorithmic bias.” (Kostick-Quenet et al., 2022, p. 93). It is important to address the implication that the authors may believe case study analysis is only effective in addressing algorithmic bias; I propose that Kostick-Quenet et al. ultimately desire to mitigate injustice and harm, as seen in “Labeling as such without broader regulatory reforms does not address the underlying justice-based concern,” (2022, p. 98) and “raise not only ethical but legal and safety concerns… can generate less accurate or reliable results for minorities” (2022, p. 92). As a result, the case study method is applicable in cases involving epistemic injustices and harms that stem from dosing inaccuracies.

Control-IQ is a closed-loop (the user does not have to manually input their glucose level as the Continuous Glucose Monitor and pump interface directly with one another) insulin dosing software that intends to assist insulin pump users in staying within range by automatically delivering dosages and fine-tuning basal rates to respond to low and high glucose levels. Control-IQ relies on a combination of basal rates (insulin that is present in an individual constantly, expressed as X.X units/hr), Carb to Insulin ratios (determines how many units of insulin are delivered based on carbs consumed, expressed as 1 unit : XX grams), and Correction Factor (how many units of insulin for every XX mg/dL above a certain target glucose level, expressed as 1 unit : XX mg/dL) to calculate automated dosages according to a study performed by Messer & Breton (2023). However, this information is only informally disclosed during pump training sessions and within this NIH study. The Control-IQ setting tab within their app only lists weight, and total daily insulin (A combination of basal rate * 24 + a sum of every additional unit delivered). This implies Control-IQ considers weight and total daily insulin as more important, yet Messer & Breton conclude “The correction factor setting may be the most impactful ‘lever to pull’ for clinicians and CIQ users to optimize time in range” (2023).

Epistemic injustices are present within this software even if it works perfectly for an individual. Firstly, the name ‘Control-IQ’ implies that individuals who are unable to control their blood sugar levels are lacking in intelligence, ignoring the reality of information overload, diversity of experiences, and the soaring cost of insulin and insulin pump supplies. This reflects a testimonial injustice, as the patient’s lived experiences are undervalued in favor of the algorithm’s logic. Furthermore, using this software prevents the user from altering their target blood glucose level, and their insulin duration, despite these statistics not being mentioned under the CIQ settings page. It is extremely likely for an individual to suffer from low/high blood sugars due to a miscalculated CIQ dose, as the Continuous Glucose Monitoring (CGM) data it relies on is impacted by the pressure of wearing a winter jacket, or the increased blood flow from weightlifting. In these cases, the responsibility of the outcome often still falls on the patients, as CIQ is difficult to hold accountable despite it making calculation errors like type 1 diabetics. Cases of poor management often stem from a lack of drug access and poorly addressed comorbidities (such as ADHD, Dementia, Alzheimer’s).

These harms could be addressed if there was greater transparency between the algorithms. The NIH study and the CIQ app unclearly imply 7 statistics (weight, total insulin, correction factor, basal rate, carb ratio, target blood sugar, insulin duration) are used in dosage calculations. It should be simple for the company to provide examples based on these 7 statistics. For example, displaying that if your sugar is 120 mg/dL and expected to be 140 mg/dL within 30 minutes: CIQ will deliver 0.5 units of insulin. This transparency would allow the user to evaluate their statistics with greater discretion. After swapping to Control-IQ, I have had to significantly alter my correction ratio, as I responded to 3 hr insulin duration, but CIQ pins it to 5 hr and assumes I am currently processing insulin when it has already been metabolized by my body. This rigidity and lack of transparency highlight a hermeneutical injustice as patients are unable to clearly interface with the decision-making process behind their insulin dosing.

Prior to CIQ, the only dosing factors I relied on were intuitive and clear: correction factor, carb ratio, and target blood glucose level. It is common practice for endocrinologists to allow patients to alter their correction factor and carb ratio without their input, as most type 1 diabetics are trusted to engage with them. In contrast, the subversive nature of CIQ calculations prevents patients from receiving an interpretable understanding of their dosing decisions. This point is further highlighted by the fact that type 1 diabetics have managed their sugars prior to the existence of CIQ and similar technologies and continue to do so.

The use of dosing algorithms transforms the traditional patient-provider relationship from a dyadic interaction into a triadic interaction involving the patient, provider, and algorithm. This new triadic relationship is not truly fair because these algorithms rely on data from healthcare providers, ultimately reinforcing provider recommendations (Kostick-Quenet et al., 2022, p. 95). This shift disrupts the balance of the patient-provider relationship by introducing an algorithmic decision-maker that is neither transparent nor accountable and strips decision-making power away from the patient.

Secret Algorithms, IP Rights, and the Public Interest explores how the opacity resulting from the shift of software being considered patented information to trade secret protections can lead to adverse outcomes, emphasizing the need for transparency to ensure accuracy and fairness (Ryan, 2020). Meghan F. Ryan states “where algorithm-based software is concerned, the law encourages trade secret protection over patent protection. This translates into encouraging algorithm secrecy over algorithm transparency.” (2020, p. 62). As a result, software developers are highly protective of the algorithms they deploy, and I presume they will continue to be secretive about the new machine learning algorithms they develop for healthcare contexts. This assumption is only strengthened by the fact that the current administration is heavily allied with large tech corporations and their data privacy interests, exemplified in sentiments from fired FTC commissioners that fear Trump will urge regulators to lay off tech donors. (Collier, 2025).

In brainstorming solutions for addressing the injustices stemming from CIQ, the first solution I came up with mirrored the solutions recommended by Kostick-Quenet et al.: increased federal oversight through the FDA and FTC (2022 p. 96) that call for improved transparency; however, I believe that this solution is no longer sustainable. The rulemaking authority of administrative agencies such as the FTC and FDA have been significantly weakened by the Supreme Court overturning the Chevron Doctrine in Loper Bright Enterprises v. Raimondo on June 28, 2024. The Chevron Doctrine allowed for administrative agencies to interpret ambiguous language in laws and required federal judges to grant judicial deference to administrative actions in those cases. The ruling in Loper Bright Enterprises v. Raimondo now allows federal judges to rule against administrative interpretations in cases involving rule-making agencies. I highly doubt Kostick-Quenet et al. expected this outcome and personally agree that rule-making agencies would have been a far more effective solution compared to the solution I have proposed, had the Chevron Doctrine not been overturned.

In the case of CIQ, the epistemic injustices can be effectively addressed by granting patients the ability to “understand and control how their health information is used.” (U.S. Department of Health and Human Services, 2022). If the conflict preventing disclosure stems from fears of IP violations and rule-making agencies are weakened in their ability to fight against tech companies, I propose requiring healthcare software companies, specifically in dosing and ML algorithmic training contexts, to offer NDAs that guarantee complete transparency once signed. This approach would balance the need to protect proprietary technology while granting patients and providers access to critical decision-making processes. In the first case, your patient health information is being used in deciding your treatment dosage, whilst in the ML training case, your patient health information is being used to train an algorithm. As a result, you have a right to understand how this information is utilized. Under the treatment exception for HIPAA, the Department of Health and Human Services has already laid out a framework for consensual disclosure:

“Individuals have the right to request restrictions on how a covered entity will use and disclose protected health information about them for treatment, payment, and health care operations. A covered entity is not required to agree to an individual’s request for a restriction but is bound by any restrictions to which it agrees.” (Office for Civil Rights (OCR), 2009).

The practice I have proposed doesn’t seek to request restrictions, and the previous quote is only meant to highlight precedent for contractual relationships between patients and covered entities. Employees of software companies also have access to algorithmic information and are subject to NDAs; if an individual’s data is being used to train machine learning algorithms, they are a non-significant aspect of the ML algorithm creation process, similar to the employee.

The case study exploration framework utilized by Kostick-Quenet et al. is concise, clear, and simple. Most importantly, it encourages individuals to come up with innovative solutions as they navigate the restrictions only found within the case. It is surprising that solutions from analyzing ethical issues through a specific case study can lead to solutions with a wider degree of applicability. For example, researchers have long understood that covariates impact algorithmic outcomes, and accounting for those is difficult. Researchers have also long understood the prevalence of systemic biases, but the LVAD case included both aspects and challenged Kostick-Quenet et al. to come up with solutions that effectively address both bias vectors and suggest innovative solutions such as developers leveraging providers to reduce their selection biases, as both the developers and the providers would prefer a highly accurate and unbiased ML algorithm (2022 p. 99). This leveraging practice can be shifted to address other vectors of harm, as developers with open-source systems may be preferred by patients and pull more customers. In the CIQ case, the restrictions of newfound developments in rule-making practices and the treatment exception under HIPAA forced me to concede stricter regulation and advocate for a patient-developer relationship that exists outside of the provider and regulatory sphere. Despite Kostick-Quenet et al. and I brainstorming solutions in unique socio-political landscapes, the practice of exploring a tool and identifying what harms it causes has resulted in creative solutions.