First, you have a false dilemma. If your users are not programmers and know only IDL, any other language (custom or existing) will look foreign to them. But that does not mean that they cannot learn something else. In fact, you would say that this is not really a problem: you'll have to write a tutorial and point them to documentation and other materials. All the languages below have good documentation. Common Lisp is a standardized language.
Lua
This is covered in another answeranother answer, but I would still recommend it. The language is made to be embedded, and works well with C++.
Python
Boost.Python is a popular way of integrating C++ code and Python. From what I heard, it seems that Python and especially NumPy is also popular for scientific computing.
Clasp
Even if you don't intend to use Common Lisp, please have a look at CLASP (video), which is an implementation of Common Lisp for C++.
A full Common Lisp implementation seems like quite a sledgehammer to crack a nut and the syntax will be quite foreign to the users (which in many ways is what I wanted to avoid by using a pre-defined language).
I see it the other way: instead of having half a language that barely works (this is not directed at above languages, but about custom ones), a full Common Lisp implementation gives you a stable and mature environment. I don't know what your scientific data is about, but your user could even work with Maxima directly from your environment, which has an infix syntax if you want.
Library/FFI
Am I looking at the problem of building a REPL-like environment from the wrong direction - should I instead build my program into a library and use an interactive language with FFI against it?
This is a possible way to do it, but I honestly cannot say which way is better for you.
