7
$\begingroup$

I'd like to obtain a symbolic result for the following integral:

Sigmoid[x_] := 1/(1 + Exp[-x])
Integrate[Sigmoid[x]*Log[Sigmoid[x*w]], {x, -Infinity, Infinity}]

The correct answer is -Pi^2 (w^2 + 1)/(12w).

Mathematica can do it for explicit small integer values of $w$ (eg, w=1 or w=2).

Improve this question
$\endgroup$
3
  • 9
    $\begingroup$ Why can't it do this? well, Maple can't. Rubi can't. Fricas can't. So it is not only Mathematica which can't. May be chatGPT can :) $\endgroup$ Commented Dec 5, 2023 at 0:24
  • 2
    $\begingroup$ This function is available under the name LogisticSigmoid. $\endgroup$ Commented Dec 5, 2023 at 4:12
  • 1
    $\begingroup$ @Nasser Trust me when I say that chatGPT can't. $\endgroup$ Commented Dec 6, 2023 at 15:53

2 Answers 2

Reset to default
17
$\begingroup$ 
$Version

(* "13.3.1 for Mac OS X ARM (64-bit) (July 24, 2023)" *)

Clear["Global`*"]

Sigmoid[x_] := 1/(1 + Exp[-x])

seq = Table[
  Integrate[Sigmoid[x]*Log[Sigmoid[x*w]], {x, -Infinity, Infinity}], 
  {w, 1, 5}]

(* {-(π^2/6), -((5 π^2)/24), -((5 π^2)/18), -((17 π^2)/48), 
  -((13 π^2)/30)} *)

Using FindSequenceFunction to generalize,

int[w_] = FindSequenceFunction[seq, w] // Simplify

(* -((π^2 (1 + w^2))/(12 w)) *)

Comparing with numeric integration,

intN[w_?NumericQ] := 
 NIntegrate[Sigmoid[x]*Log[Sigmoid[x*w]], {x, -Infinity, Infinity}]

Plot[{int[w], intN[w]}, {w, 0, 10},
 PlotStyle -> {Automatic, Dashed}]

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for showing me how I could have done this. I guess I did the same thing, manually: doing numerical approximations in C++, plotting those and then guessing what the underlying function is. I did Sigmoid[x]*Log[Sigmoid[x*w + b]] that way too in the meantime (note the extra +b). $\endgroup$ Commented Dec 6, 2023 at 15:56
  • $\begingroup$ Thank you @Bob for this tool. $\endgroup$ Commented Aug 31 at 12:18
0
$\begingroup$

So, I don't have a full solution, but I just want to log what work I've managed so far.

Intuitively I just randomly tried breaking the integrand down using the (even / odd ) function transform f(x) -> (f(x)+f(-x))/2 + (f(x)+f(x))/2. This makes the numerator much cleaner looking. The even part integrates easily. The odd part really breaks down into this integral:

Integrate[Log[1 + E^(-w x)]/(1 + E^x), x]

or with a substitution:

Log[1 + u^w]/(u (1 + u))

This is the simplest version of your problem I could find. My caclulus skills aren't exceptional, but I couldn't get any techniques to handle this that I know (tried differentiating under the integral sign wrt to w, but this doesn't work), so I'd probably bring this to the math stackexchange after searching for a bit.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My guess is that the integral itself is everything but simple. Mathematica loves to produce lots of PolyLog terms in the result. It's just that the integral from -inf to +inf becomes remarkably simple again. $\endgroup$ Commented Dec 12, 2023 at 13:18
  • $\begingroup$ This is the context in which I needed this / the work that I did that led to this: github.com/CarloWood/machine-learning/blob/… Then on line 521 the result for x*w + b. Turns out that all I did was derive in my own way the binary cross-entropy loss function :) $\endgroup$ Commented Dec 12, 2023 at 13:30

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.