Understanding the SIGMOID Activation Function

 

SIGMOID Activation Function

When we are building a neural network almost always the function we want to model is not linear in nature. To deal with this we need to use an activation function which introduces non-linearity into the neural network. A common activation function uses the SIGMOID function, define by the equation below.

For a given value of x the SIGMOID function will produce an output between 0 and 1. We can demonstrate this with the following R code.

library(ggplot2)

 

# Define sigmoid function

sigmoid <- function(x) {

  1 / (1 + exp(-x))

}

# Generate data

x <- seq(-10, 10, length.out = 100)

y <- sigmoid(x)

 

# Create data frame

df <- data.frame(x = x, y = y)

 

# Plot sigmoid function

ggplot(df, aes(x, y)) + geom_line() + labs(title = "Sigmoid Function", x = "x", y = "y")

Adding a weight to the SIGMOID

By adding a weight W to the sigmoid equation we can vary the gradient of the slope between 0 and 1, as shown but the R code below.

library(ggplot2)

 

x <- seq(-10, 10, length.out = 1000)

 

# Define sigmoid function

sigmoid <- function(x) {

  1 / (1 + exp(-x))

}

 

# Generate sigmoid curves for different constants

constants <- c(0.5, 1, 1.5, 2)

curves <- lapply(constants, function(c) sigmoid(c * x))

 

# Plot the curves

ggplot() +

  geom_line(aes(x, curves[[1]], color = "0.5")) +

  geom_line(aes(x, curves[[2]], color = "1")) +

  geom_line(aes(x, curves[[3]], color = "1.5")) +

  geom_line(aes(x, curves[[4]], color = "2")) +

  ggtitle("Sigmoid Function with Varying Slopes") +

  xlab("x") +

  ylab("y") +

  scale_color_manual(values = c("0.5" = "blue", "1" = "red", "1.5" = "green", "2" = "purple")) +

  theme_bw()

Bias connection

We can add another input to the activation function called bias input. Bias input is always one multiplied by a weight b. The purpose of the bias input is to move the sigmoid function either to the left or to the right as by the R code shown below.

sigmoid <- function(x, bias) {

  1 / (1 + exp(-x + bias))

}

 

# Set up plot

plot(NULL, xlim = c(-10, 10), ylim = c(0, 1), xlab = "x", ylab = "y")

 

# Plot sigmoid curves with different biases

curve(sigmoid(x, bias = -3), add = TRUE, col = "blue", lwd = 2)

curve(sigmoid(x, bias = 0), add = TRUE, col = "red", lwd = 2)

curve(sigmoid(x, bias = 3), add = TRUE, col = "green", lwd = 2)

 

# Add legend

legend("topleft", legend = c("-3", "0", "3"), col = c("blue", "red", "green"), lwd = 2)

Logistic Regression

The Sigmoid function can be used to calculate logistic regression. This kind of regression is used to calculate the probability of a binary outcome or decision based on the input.

The Paradox of Time Management: Why Slowing Down and Being Choosy is the Key to Success

 

Introduction

I was thinking about time management, and I remembered a concept that was on the tip of my tongue, but I could not quite articulate it. This led me on a search, and what I discovered were some notes I made in May of 2022 from reading the book 4000 hours by Oliver Burkeman. The following is my attempt to flesh out my notes into something more readable. But if you have time, I urge you to read the book, it is excellent. 

Summary

• It is natural to never get to the end of your to-do list.
• Slow down.
• Be discerning about the tasks we do.
• We need to be more choosey about what we take on.
• Celebrate not being able to do it all and say no to most things.
• What we achieve is down to chance.
• Pay yourself first. 

Delusion

Are we deluding ourselves? Will we ever get to the bottom of our to-do lists?  We all have a finite life and as a society we are trying to cram in more and more, we aim to be highly efficient and be as productive as possible. This goal of becoming our most efficient and effective self’s is draining the joy out of life. 

It seems to me that the more we tick off our to-do lists the bigger and not smaller our lists get. This paradox tells us that we need to admit defeat and acknowledge we will never get everything on our to-do list done. If you had a fairy godmother who could magically disappear all the things on your to-do list, then miraculously by the end of the week your list would be just as full as it was before the magical intervention occurred. What we need to do is change our relationship with the things we have on our lists. 

Slowing down and being more discerning

Time is an abstract concept. We split our life up into buckets of time that do not actually exist. This process of putting time into buckets is the source of our stress. A large number of us measure our self-worth by the number of things we can tick off each day. This act of speeding through our lists is an act of avoidance, we are trying to ignore our fears. 

But it is impossible to master time, we will never complete everything on our lists and most of the tasks we complete aren’t going to make any difference to the grand scheme of things. Taking control of your list is not done by speeding up, but by slowing down and being more discerning about what we do. 

Oliver Burkeman said that the paradox of limitation is the more you try to manage your time to get the feeling of total control, the more empty and frustrating life gets. But the more you confront the fact we have limited time and work with it the more meaningful our lives can become. 

The only power is choice.

Busy, we all have too many things to do than we can fit into the available time. Not being able to fit it all in is making us feel overwhelmed. To make matters worse we should not feel guilty for not fitting in everything, but we do. 

Efficiency is a trap, the better our productivity system is at giving us time the more we fill it with more and more tasks. This is the effect known as Parkinson’s law, our jobs expand to fill the available time. All this means is we live under the illusion that one day we will get everything done. 

The only thing we can do is choose. We can only choose to do a few important things and deal with the fact that the other things will not get done. Our life is finite and this means we can only do a finite amount of things. Our only power is choice.   

Celebrate finitude.

I like the word finitude. It reminds us to make the most of our time and appreciate the fleeting moments that we have. Finitude is the quality of being finite or being limited by bonds or scope. This means that every time we make a choice to do something (or not do something) we define who we are. 

If finitude did not exist and life went on forever, there would be no need to make any choices and life would stop having meaning. The fact that we cannot do everything is therefore something to be celebrated because it makes us unique and gives us a reason to make the most of our finite time. 

Get better at not doing things.

The point of a time management system is not to get better at doing things, but to get better at not doing things. If we accept we can not do it all, then we need to stop acting as if we can. We need to stop feeling guilty about what is not getting done.

We need to commit to a few things and achieve them. This needs to be done in a way that does not leave us riddled with guilt. The three best strategies for doing this are:

·        Pay yourself first: Schedule time to get your pet projects done first.

·        Limit your work in progress: Only have three active tasks open at any one time.

·        Resist middle priorities: Focus on your top five priorities and ignore the rest. 

Enjoy Boredom.

Stop wasting time. We are all easily distracted, but the distractions are stealing our time. In the end watching the cat video on YouTube will not make us more fulfilled, this type of activity will just waist precious moments of our life that we can never get back.

 What we focus on will change how we experience it. This means if we focus and work harder at something we find boring in the end we will come to love it. Don't resist the pain of being bored, work through it. Ironically sometimes the solution to boredom is just to accept that sometimes we will get bored, live with it. 

It's all down to chance.

Hofstadter law - It always takes longer to do something even if you take into account Hofstadter law. This is a conundrum! It means it is good to plan but ultimately our plans are meaningless. In fact, our plans are a way of giving certainty to what can never be achieved. 

We do not have time, we are time. Our desire for certainty is not something we can choose. What we ultimately achieve is all down to chance. The outcome is outside of our control. We should therefore enter tasks with the mindset of not really caring what happens in the end. 

Live for today

We all know we have a finite amount of time. So why do the groundwork for the future, when the future might not even come, and our plans are no better than chance. We should stop living for the future and focus on the now. We shouldn't study for a degree just to get somewhere in the future. We should do it because in the here and now it brings us to life and determines who we are. What happens in the future is then of no consequence.

 We should treat every moment as if it is the last time, we get to do something. 

Conclusion

In the end what you do with your life doesn't really matter. You can be sure the universe doesn't care one single bit. Every life is ordinary, and this is good to remember when we feel things are getting too much for us. We can use this thought as a kind of therapy to keep us sane when we face the constant onslaught for our time and focus.

 Focus on the things you can do now and only on the things you can do now. Finitude is the only problem we have when you think about it. Just focus on the next thing you must do right now. We don't need security we just need to focus on the few things that count right now. 

References

·        Four Thousand Weeks: Embrace your limits. Change your life. Make your four thousand weeks count. (Audio Download): Oliver Burkeman, Oliver Burkeman, Penguin Audio: Amazon.co.uk: Audible Books & Originals

·        Parkinson’s law: Parkinson's law - Wikipedia

·        Hofstadter's Law:https://en.m.wikipedia.org/wiki/Hofstadter%27s_law

·         

What is ‘e’

Just like pie, ‘e’ is widespread in mathematics and shows up in many different places. In this post I want to explore a little about what ‘e’ is and look at how you might draw a graphical representation of it using the R programming language.

Try this.

Before we begin exploring ‘e’ try this simple exercise using a calculator:
1. Enter a memorable 7-digit number on your calculator.
2. Take the reciprocal of that number, by pressing the 1/x button.
3. Add 1 to your answer.
4. Now raise the number to the power of the original 7-digit number.

Does your answer begin 2.718? If the answer is yes then you have found ‘e’!

What is ‘e’

$$e=lim_{n\to\infty}(1+\frac{1}{n})^{n}$$ We define ‘e’ to be the number that $(1+\frac{1}{n})^{n}$ is getting closer and closer to as ‘n’ gets larger and larger. $$ e=2.718281828459045... $$ If we replace 1/n with x/n we get the following equation: $$e^{n}=lim_{n\to\infty}(1+\frac{x}{n})^{n}$$ This formula $e^{n} $ has lots of interesting properties and applications.

R code to Plot ‘e’

With the simple programme below, we can show you what the growth of $e^{n}$ looks like.

x <- seq(-5, 5, by = 0.1)
y <- exp(x)
plot(x, y, type = "l", xlab = "x", ylab = "e^x")

This produces the following graph:

$$lim_{n\to\infty} e^{n}=\infty \\ lim_{n\to-\infty} e^{n}=0 \\ lim_{n\to0} e^{n}= 1 $$ The graph shows at minus Infinity the y axis approaches 0 and at Infinity the y axis approaches Infinity. But the y axis always passes through a value of 1 when x is 0.

R code to Plot inverse ‘e’

We can also plot the graph for the inverse of ‘e’ as follows:

x <- seq(-5, 5, by = 0.1)
y <- exp(-x)
plot(x, y, type = "l", xlab = "x", ylab = "e^-x")

$$ lim_{n\to\infty} e^{n}= 0\\ lim_{n\to-\infty} e^{n}= \infty \\ lim_{n\to0} e^{n}= 1$$