How might mathematics have been different?

I think the notion of topological space could have very easily not existed - or at least be a lot more of a marginal notion.

Of course, there would probably still have been some notions of "space" and "continuous function". But there are many alternatives to our traditional concept of topology that could have been used instead, and I feel like once one of these becomes standard there are no strong incentives to really develop another.

• There are many non-equivalent point-set notion of spaces, like convergence spaces, condensed/pyknotic sets, sequential spaces, compactological spaces, the sort of compactly generated spaces used to get convenient categories of spaces, etc...

• There could be no "general" notion of spaces, but lots more specific notions that we use in different contexts (manifolds, analytic sets, CW-complexes, pro-discrete spaces, ...)

• A philosophical aversion to consider the continuum as a collection of isolated points could have led to something more like our pointfree topology (where open subspace and a notion of locality are taken as primitive independently of an underlying set of points) introduced before point-set topology and become the dominant notion.

• There are probably some other alternatives we haven't even considered.

We can only speculate how our modern mathematics could be different. But there is one case which can be studied "empirically". I mean there is at least one historic example of two different ways mathematics can be developed.

I mean the difference between Greek and Babylonian mathematics. They developed at approximately the same epoch in two very different civilizations. While early Greek mathematics concentrated on geometry and rigorous proofs, Babylonian mathematics concentrated on computations and algorithms. In particular they invented a positional system, including the notation for zero. But did not invent "axiomatic method", calculus and advanced geometry.

One of the main applications of mathematics in both civilizations was to astronomy, and here we have much more documented material. There is a well developed Babylonian mathematical astronomy which is completely different from what we know from Greece. Despite the fact that after 300bc there was a substantial interaction between the two, Babylonian astronomy survived as a separate science until 19 century (in Tamil astronomy in India). There is an enormous difference in how both systems treated the same phenomena, for example Lunar and Solar eclipses. And both systems were successful to some extent. Probably one cannot say that Babylonian methods were very "inferior" to Ptolemy in predictive power.

Refs. O. Neugebauer, The exact sciences in antiquity, Princeton, 1962.

B. L. van der Waerden, Ontwakende Wetenschap (Science awakening), Groningen 1950, there are English translations.

Nowadays, we associate mathematical rigor with symbolic/algebraic reasoning. Geometric reasoning is considered rigorous only insofar as it can be backed up with symbolic/algebraic reasoning, and "picture proofs" are regarded with suspicion.

However, one can imagine an alternate universe in which the script is flipped. After all, for centuries, the paradigmatic example of mathematical rigor was Euclidean geometry. In A formal system for Euclid's Elements, by Avigad, Dean, and Mumma, it is shown that the diagrammatic reasoning in Euclid can be made completely rigorous. So perhaps one can imagine a world in which geometry continues to be the ultimate foundation of mathematics, with increasingly strong logical systems being characterized by increasingly strong geometric axioms, rather than increasingly strong set-theoretic axioms.

Joe Shipman has argued for an alternative history in which people came to accept the existence of a countably additive "measure" on all subsets of the reals. This "measure" would not be translation-invariant, of course. But the axiom has some nice consequences, including strong Fubini theorems. In this hypothetical world, the negation of the continuum hypothesis would be regarded as a theorem.

I'm not sure how plausible I think Joe Shipman's scenario is, but there are other scenarios surrounding the axiom of choice that one could imagine. People could have come to accept "all sets are Lebesgue measurable" along with the axiom of dependent choice. In this scenario, measure theory would better match probabilistic intuition (there would be no Banach–Tarski paradox, for example, because full AC would be rejected).

Descriptive set theory was essentially born in 1917 because Souslin noticed a mistake in a 1905 paper of Lebesgue. To be more precise Lebesgue "proved" that the projection to $\mathbb R$ of a Borel set in $\mathbb R^2$ is Borel (his mistake was in assuming that decreasing countable intersections commute with projections).

It's hard to guess exactly what would have happened if Lebesgue had noticed this issue himself, but the whole field of descriptive set theory could have developed in a different way from the one we know.

This has been talked about before on MathOverflow, but I suspect that there's a fair amount of historical contingency in the handling of size issues in category theory. In particular, I could imagine an alternate history in which the amount of effort put into addressing size issues is roughly the same as it is now but in which the communities of fields that regularly use inaccessible cardinals (i.e., Grothendieck universes) are willing to confidently assert that their results go through in $\mathsf{ZFC}$.

Part of the reason I suspect this is that this is essentially the case with size issues in model theory. Model theory becomes moderately easier under the assumption of the existence of many inaccessible cardinals, because it allows one to build saturated models of arbitrary theories. Despite this, model theorists culturally do not like assuming the existence of inaccessible cardinals and instead (sometimes implicitly) rely on certain approximations of inaccessible cardinals (and occasionally a more specific approximation of saturated models called 'special models').

The way I would envision this working is that (analogously to the situation in model theory), one would pick some cardinal $\kappa$ that they're reasonably sure is going to be larger than all of the small objects they actually care about and then work in categories derived from or closely related to the category $\def\Set{\mathrm{Set}}\Set_{\beth_{\kappa^+}}$ of sets of size less than $\beth_{\kappa^+}$, which is closed under limits and colimits of size at most $\kappa$ (since $\kappa^+$ is a regular cardinal and $\beth_{\kappa^+}$ is a strong limit cardinal). If one finds that they need a bigger 'little universe', they could then pass to the category of sets of size at most $\beth_{(\beth_{\kappa^+})^+}$ and then to $\beth_{(\beth_{(\beth_{\kappa^+})^+})^+}$ and so on.

Obviously, just like with model theory, there is an (in my opinion pretty small) amount of extra conceptual burden here, in that $\Set_{\beth_{\kappa^+}}$ has two notions of 'smallness' (i.e., $\leq \kappa$ and $< \beth_{\kappa^+}$) rather than the single notion of smallness one gets with an inaccessible cardinal. And I think that on some level this explains the cultural difference with regards to size issues between model theory and fields outside of logic. Many of the people working in model theory in the early days were pretty familiar with set theory and would have been fine with the occasional cardinal arithmetic calculation needed to make the above kind of reasoning work carefully, and (in part because of this) most model theorists today are comfortable handwaving these kinds of size issues entirely while asserting that their theorems are theorems of $\mathsf{ZFC}$, even if they aren't particularly comfortable with set theory. This seems analogous to me to the way many people often handwave the precise bookkeeping of universes entirely, which is why I feel like the assertion in my first paragraph is reasonable.

Now, it should be said that I also don't think that this matters that much. I frequently use much larger large cardinals in my own papers. And while I do think that $\mathsf{ZFC}$ is a more natural theory than people seem to sometimes think, I also don't think its particular consistency strength is all that special.

In principle, one might be worried about 'universe creep' (i.e., the gradual increase in the number of universes one regularly assumes) as a result of building on earlier work. But in the 95 years since Sierpiński and Tarski and separately Zermelo wrote down the definition of (strongly) inaccessible cardinals, the extent of this creep in mathematics outside of set theory seems to be at most $3$ to $5$ universes. (If I recall correctly, somewhere in the work of Jacob Lurie he explicitly assumes $3$ universes. Moreover, in a recent conversation on the Lean Zulip, Mario Carneiro calculated the maximum number of universes actually used in Mathlib and it was somewhere between $3$ and $5$ depending on how one counts. I also just don't think I've ever seen the explicit assumption of an explicit finite number of universes greater than $3$ in a paper.) As someone who regularly interacts with set theorists, this is an amusingly weak assumption to me given that even a Mahlo cardinal (i.e., an inaccessible cardinal $\kappa$ such that the set of inaccessible cardinals smaller than $\kappa$ is stationary) is regarded by set theorists as puny and the first 'actually big' large cardinal is regarded as being something like a measurable.

If computer science and computability were a thing much earlier, we maybe would use computable numbers instead of the reals in many places, since they have the advantage of being a countable set containing all "interesting reals" like roots, $\pi$, $e$ and so on.

Modern calculus as we know it is all about real numbers and the functions between them. However in the times of Newton and Leibniz, it was all about the change of quantities in relation to the change of other quantities, often involving geometry (Interestingly this viewpoint is still quite prevalent in the way physicists understand their calculations). It is only in the centuries afterwards that the viewpoint shifted.

I would blame a good part of that shift to people having a good understanding of numbers, developed in medieval times through countless classic texts on calculations, elementary algebra and crucially by adopting the Arabic number system, allowing for things like decimal points in favor of fractions. Another simple, but crucial idea was the Cartesian coordinate system, turning geometry into number problems.

Now people often argue that Archimedes was another founder of calculus. He certainly had a clear understanding of integration, in particular in terms of volume and area. So an alternative history, where he or one of his contemporaries also discovers differentiation and the fundamental theorem is not too far fetched.

But then since algebra and a flexible number system are missing, this inverted order of discovery would have lead to centuries more of geometric, "quantity" focused development of calculus. People might have even come up against the limits of that approach (e.g. the questions of differentiability and "what is a function") in a completely different way and solved them using different tools.

The solution to FLT was historically contingent. If Frey had not made his observation about a counterexample to FLT leading to a possible counterexample to modularity of elliptic curves over $\mathbf Q$, which was put into a precise form by Serre and then proved by Ribet, Wiles would not have been inspired to spend $7$ years trying to prove modularity. In the BBC documentary on FLT, Coates said the general feeling in the 1980s was that modularity of elliptic curves over $\mathbf Q$ would not be proved “in our lifetime”.

So arguably no Frey observation linking a counterexample to FLT with unusual elliptic curves probably would have meant no solution to FLT in the $1990$s, which would have made the announcement in $2012$ about a solution to the $abc$ conjecture take on an additional significance because the $abc$ conjecture implies FLT for all large exponents (nonconstructively), making the interest in a more explicit form of $abc$ more compelling.

[Edit] Another example is the Weil conjectures as the reason for Grothendieck��s work in algebraic geometry. Logically speaking you don’t need the Weil conjectures in order to develop modern algebraic geometry, but it is nevertheless true that solving those conjectures was the long-term goal inspiring Grothendieck (just as FLT was for Wiles). The comment by @algori to Shulman’s post imagines a hypothetical past where Grothendieck never worked in algebraic geometry. If there had been no Weil conjectures, then that SF past is much more likely, which would have made some areas of math very different today.

KConrad mentioned FLT in an answer, in the context of the late XXth century. But its rôle in the development of number theory, commutative algebra and algebraic/arithmetic geometry has been, one might argue, enormous starting from its very first appearance. And I think it might well happen that if Fermat would not, by whatever reason, write his famous

... cuius rei demonstrationem mirabilem sane detexi. Hanc marginis exiguitas non caperet

the whole extraordinarily rich cascade of breakthroughs in these fields might well be delayed for many decades, and the current shape of those domains of mathematics might quite well become entirely different from what we have now.

I keep wondering why probability theory, measure theory, and (naive) set theory did not develop earlier, especially in comparison to algebra, consider, for example, Galois theory. When I learnt Galois theory, I was surprised how old the results were. On the other hand, when I was tought the law of large numbers from pairwise independent identically distributed random variables, I was suprised how recently the result had been obtained. I find this especially surprising as probability theory seems closer to applications than algebra and unlike in the case of physics one cannot argue so easily that one had to wait for experimental technology to catch up.

Somehow the notion of a function (between sets) play an important role and have contributed to forming concepts in mathematics. Sometimes I wonder how mathematics would have evolved if we used relations instead of functions.

Often you see proofs that under certain assumption something is well defined and they you define a function mapping something to that element. If the assumptions do not hold, then you can't continue to work. In the setting of relations, we would then map the source element to all of the choices.

I would like to propose another alternate history, this time about set theory. Set theory has, so to speak, as most things, two parents, analysis and logic. The impression I got was that the developments in these areas happened quite independently. Mathematicians were able to do number theory for millenia without resorting to axioms. In the case of set theory, they did it not even for half a century. I am afraid that I do not know enough about the history of geometry before Euclid or in other intellectual traditions to give an exposition here. But I find it perfectly imaginable that set theory could have been practised "naïvely" for a considerably longer period than it historically has been. Suppose that we are in the present, i.e. the year 1950, set theory is practised for the better part of two centuries but academics are not that interested in logical questions and Gödel stuck to studying physics. Then the continuum hypothesis would be regarded more like the question whether there are any odd perfect numbers, old, interesting, but it not having been resolved would not be regarded as an embarrassment to the field to any further extent than in the case of the existence of odd perfect numbers and number theory. I would stop short of claiming that set theory could have had a naïve period quite as long as number theory but one should keep in mind that we have seen a dramatic increase in the efforts spent on science and mathematics in the period discussed which distorts the comparison to number theory a bit. I believe that only a few things would have to have been a little bit different, a greater diversity in people influencing the field at an early stage, a closer relationship to analysis, less of an obsession with attaining a complete theory and probably a few more things which I fail think of at the moment.