Within a simulated payment system, transaction-scoring tools flag potential issues, which analysts then evaluate individually to determine legitimacy. A key safeguard exists: every alert undergoes human review before any action is taken.
This review, as a documented record, can reveal certain aspects to someone reviewing it later, while leaving other aspects ambiguous. For instance, consider a hypothetical payments platform where its software generates around 500 alerts each day.
An analyst reviews each alert and then closes it, marking it either as a false alarm or escalating it. Over a hypothetical 30-day month, the log contains 15,000 closures, with 14,100 of them labeled as ‘false alarm.’ This amounts to 94 percent.
What the log shows
This figure can be interpreted in two ways. The software might generate numerous alerts that are truly insignificant, and the analyst’s decision to dismiss them could be correct. Alternatively, the analyst might be closing alerts more quickly than they can be thoroughly examined.
Both interpretations result in the same log entry. The closure status and timestamp alone do not indicate whether the customer’s history was reviewed, whether the other party was investigated, or whether the pattern was compared to previous cases.
Two analysts, one thorough and one in a hurry, can leave identical traces. This makes it difficult to determine the quality of the review based on the log alone.
A way to look behind the rate
One method to go beyond the percentage is to conduct a test with a predetermined outcome. On a specific day within a hypothetical week, a colleague inserts 20 alerts into the queue. These alerts are adapted from previously identified irregularities in closed investigations, but with identifying details altered.
They are blended with the regular alerts without any special marking, making the total queue size 500 alerts. The analyst processes them at their normal speed. In this simulated scenario, 14 out of the 20 known alerts are escalated, along with 31 of the 480 standard alerts.
Read Also: AI Helps African Finance Functions Leapfrog
Each figure addresses a distinct question. The first figure, 14 out of 20, indicates how much of the known material was correctly identified. The second figure, 31 out of 480, shows how many other alerts were forwarded for further review.
What a rehearsal cannot say
Twenty known cases give only a limited estimate. A second day could produce another result for reasons that have little to do with attention: which cases were picked, how busy the queue was, or chance.
The known cases are also derived from past events, and new patterns might differ from older ones. This test describes the performance of this specific analyst with this particular queue on this specific day. It does not apply to other teams or any real-world platform.
For the people most affected by this development, the ability to understand why an alert was closed is important. If an organisation chooses to record the reasons behind each closure, it could provide a way to understand the thought process behind the analyst’s decision, even after they have moved on.
This is particularly important when the scoring model has been updated or when a regulator requests an explanation. Without this additional information, the log may show who closed an alert and when, but it may still fail to clarify the reasoning behind the decision.
None of this says how any organisation ought to run its alert queues. Volumes, risks and duties differ, and how much recorded reasoning makes sense is a judgement for the people responsible, informed by the rules that apply to them.
Three points remain after the reading above, and each can be considered for any setup: what exactly the analyst was shown and what was not; what a later authorised reader could recover from the record; and whether anyone has looked at the review itself, and not only at the software that feeds it.
Where the record names the analyst, a closure shows who decided. Whether the decision can be explained afterwards is a separate matter, and one that can be looked at.
