Perhaps the most widely touted of GPT-4’s at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam. This paper begins by investigating the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that indicate that OpenAI’s estimates of GPT-4’s UBE percentile are overinflated. First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers (...) who failed the July administration and score significantly lower than the general test-taking population. Second, data from a recent July administration of the same exam suggests GPT-4’s overall UBE percentile was below the 69th percentile, and $$\sim$$ ∼ 48th percentile on essays. Third, examining official NCBE data and using several conservative statistical assumptions, GPT-4’s performance against first-time test takers is estimated to be $$\sim$$ ∼ 62nd percentile, including $$\sim$$ ∼ 42nd percentile on essays. Fourth, when examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to $$\sim$$ ∼ 48th percentile overall, and $$\sim$$ ∼ 15th percentile on essays. In addition to investigating the validity of the percentile claim, the paper also investigates the validity of GPT-4’s reported scaled UBE score of 298. The paper successfully replicates the MBE score, but highlights several methodological issues in the grading of the MPT + MEE components of the exam, which call into question the validity of the reported essay score. Finally, the paper investigates the effect of different hyperparameter combinations on GPT-4’s MBE performance, finding no significant effect of adjusting temperature settings, and a significant effect of few-shot chain-of-thought prompting over basic zero-shot prompting. Taken together, these findings carry timely insights for the desirability and feasibility of outsourcing legally relevant tasks to AI models, as well as for the importance for AI developers to implement rigorous and transparent capabilities evaluations to help secure safe and trustworthy AI. (shrink)

. Decision-making under uncertainty is sometimes investigated as a homogeneous problem, independently of the type of decision-maker and the level and nature of the decision itself. However, when the decision-maker is a public authority, there immediately arise problems additional to those that concern any other decision-maker. This is not always clearly recognised in orthodox discussions on decisions under conditions of uncertainty. This article investigates the methodological, strategic and procedural challenges of taking public decisions in such conditions. It focuses mainly on (...) decisions involving urban contexts, such as planning decisions regarding land use and building transformations, by trying to develop some pioneering research studies in this field. (shrink)

One of the main challenges for computational legal research is drawing up innovative heuristics to derive actionable knowledge from legal documents. While a large part of the research has been so far devoted to the extraction of purely legal information, less attention has been paid to seeking out in the texts the clues of more complex entities: legally relevant facts whose detection requires to link and interpret, as a unified whole, legal information and results of empirical analyses. This paper presents (...) an ongoing research that points in this direction, trying to devise new ways to support public prosecutors in assessing the dangerousness of individuals and groups under investigation, an activity that precisely relies on the cross-sectional evaluation of legal and empirical data. A knowledge mining strategy will be outlined that lines up, into a single metaheuristic model, information extraction, network-based inference, machine learning and visual analytics. We will focus, in particular, on the integration of graph-based inference and machine learning methods used both to support classification tasks and to explore new forms of man-machine cooperation. Experiments made involving public prosecutors from the Italian Anti-Mafia Investigation Directorate and using data from real investigations have not only shown the potentialities of our approach but also offered an opportunity to reflect on the role we could assign to AI when thinking about the future of legal science and practice. (shrink)

We present an algorithm that can anticipate tax evasion by modeling the co-evolution of tax schemes with auditing policies. Malicious tax non-compliance, or evasion, accounts for billions of lost revenue each year. Unfortunately when tax administrators change the tax laws or auditing procedures to eliminate known fraudulent schemes another potentially more profitable scheme takes it place. Modeling both the tax schemes and auditing policies within a single framework can therefore provide major advantages. In particular we can explore the likely forms (...) of tax schemes in response to changes in audit policies. This can serve as an early warning system to help focus enforcement efforts. In addition, the audit policies can be fine tuned to help improve tax scheme detection. We demonstrate our approach using the iBOB tax scheme and show it can capture the co-evolution between tax evasion and audit policy. Our experiments shows the expected oscillatory behavior of a biological co-evolving system. (shrink)

Unlike usual real graphs which have a low number of edges, we study here a dense network constructed from legal citations. This study is achieved on the simple graph and on the multiple graph associated to this legal network, this allows exploring the behavior of the network structural properties and communities by considering the weighted graph and see which additional information are provided by the weights. We propose new measures to assess the role of the weights in the network structure (...) and to appreciate the weights repartition. Then we compare the communities obtained on the simple graph and on the weighted graph. We also extend to weighted networks the amphitheater-like representation of this legal network. Finally we evaluate the robustness of our measures and methods thus taking into account potential errors which may occur by getting data or building the network. Our methodology may open new perspectives in the analysis of weighted networks. (shrink)