In the previous post, we talked about multitasking, how it relates with our daily life in a form of concurrency and parallelism, and finally, we showed why software uses them and how Java handles those two concepts.

However, the current Java thread model has a flaw regarding scalability.

In this post, we are going to see why the thread model has a flaw, how Java chose to fix it using Thread Pools.

TRY IT YOURSELF: You can find the source code of this post here.

Java Concurrency Series

• Part 1: Concurrency and Parallelism
• Part 2: Thread Pools and Executors (You are here)
• Part 3: Futures, Parallelism and Project Loom

If you like this post and are interested in hearing more about my journey as a Software Engineer, you can follow me on Twitter and travel together.

A Flaw on the Java Thread Model

As we saw in the previous post, one Java thread maps to one kernel thread, as we see in the following diagram:

This approach has a downside, Java will be bounded by the capacity of the kernel threads, meaning, if there are just 1000 kernel thread available, Java will be able to use only 1000 Java threads. Also, kernel threads are expensive to create and consume a lot of memory which Java cannot control at all.

If you use one thread to do a task, and you only have 1000 threads, you can only do 1000 tasks at a time, that’s a great limitation. Also, you don’t know how many kernel threads are available from Java, meaning, if you create more threads than the available ones Java could break.

To handle some of those challenges, Java offers thread pools.

Thread Pools

A pool is a set of elements you save somewhere to use later, it is pretty similar to a cache, with some differences regarding how you access the elements inside. The cache defines a key by each value inside the pool, so, you can access the values if you know the keys.

In the case of a pool, we define other way to access the values, in the case of a thread pool, we have a set of threads inside the pool, and the pool chooses one of them when that thread is free, meaning, the thread doesn’t have an associated task, as we see in the following images:

First, we have a set of tasks we want to process in a thread pool, those tasks usually are grouped in a queue, so, the first in first out (there are other strategies). That queue asks to the pool for a free thread.

Now, the thread starts to process the task. Meanwhile, that thread is out of the pool and cannot be used by other task until the first task finishes.

When the thread pool is empty because it needs to process too much tasks, the tasks queue must wait until another thread is free. We can set up some pools to create a new thread if the pool is empty, and do that until a defined amount to about boundlessness pools.

After a thread finishes with his tasks, it returns to the pool and is available to grab another task.

Finally, the tasks that was waiting for a thread, grabs the free one and starts execution.

NOTE: Take into account that the tasks could finish concurrently, which means, the pool might not be empty because the threads are switching between running and free pretty fast.

Now, why thread pools matter in Java? well, thread pools help us to solve Java heavy threading model, as we can reuse threads, we avoid creating heavy threads that maps with heavy OS threads. However, there are consequences:

• If the thread pool is empty, the tasks need to wait until one is available, causing contention.
• As there are not great amount of threads you can use, if you have multiple thread pools in your application, you should count all of them and make sure you won’t overload the system.
• Having long running tasks over a thread pool is not a good idea, as that thread won’t come back to the pool soon.

Now, let’s see how to use Thread Pools in Java.

Thread Pools in Java: ExecutorService

Java offers Executors: Thread pools to submit tasks using different strategies for the pool. First, let’s see a use case where we need to run 100 concurrent tasks using a pool:

  private void submitTasks(ExecutorService executor)
          throws InterruptedException {
    Set<String> threadNames = new HashSet<>();

    for (int i = 0; i < 100; i++) {
      int finalI = i;

      executor.execute(() -> {
        System.out.println(
                "Running: " + finalI + " Thread: " + Thread
                        .currentThread().getName());

        threadNames
                .add(Thread.currentThread().getName());
      });
    }
    Thread.sleep(2000);

    System.out.println();
    System.out.println(
            "# Threads: " + threadNames.size());
    System.out.println("Threads: " + threadNames);
  }

There, we receive the ExecuteService interface and use it to execute 100 tasks. We print the task number and the thread name, also, we save in a set the name of the thread which execute the tasks and print the names and the amount of threads we use at the end.

Now, let’s see the thread pool strategies:

Single Thread

We can create an ExecutorService with only one single thread. This case is useful when you know you won’t need more threads to handle the tasks, or you need to guarantee the order of execution of those tasks:

  @Test
  public void singleThreadPool()
          throws InterruptedException {
    ExecutorService executor = Executors
            .newSingleThreadExecutor();

    submitTasks(executor);
  }

Running this test we got:

Running: 0 Thread: pool-1-thread-1
Running: 1 Thread: pool-1-thread-1
Running: 2 Thread: pool-1-thread-1
Running: 3 Thread: pool-1-thread-1

........

Running: 98 Thread: pool-1-thread-1
Running: 99 Thread: pool-1-thread-1

# Threads: 1
Threads: [pool-1-thread-1]

There, we see that all the tasks run over the same thread, besides, the tasks run in order, as there are not any concurrent execution going.

FIXED POOL

We can create an ExecuteService with a fixed thread pool. In this case, we control how many threads we have. The pool cannot create more or have less threads. A good use case for this pool is when you don’t want to overload the system creating unnecessary threads and you are sure about how many you need for your concurrent load.

  @Test
  public void fixPool() throws InterruptedException {
    ExecutorService executor = Executors
            .newFixedThreadPool(5);
    
    submitTasks(executor);
  }

Running this test we got:

Running: 3 Thread: pool-1-thread-4
Running: 2 Thread: pool-1-thread-3
Running: 0 Thread: pool-1-thread-1
Running: 1 Thread: pool-1-thread-2
Running: 6 Thread: pool-1-thread-4
Running: 5 Thread: pool-1-thread-3
Running: 7 Thread: pool-1-thread-1
Running: 9 Thread: pool-1-thread-3

.....

Running: 96 Thread: pool-1-thread-4
Running: 97 Thread: pool-1-thread-4
Running: 98 Thread: pool-1-thread-4
Running: 99 Thread: pool-1-thread-4
Running: 4 Thread: pool-1-thread-5
Running: 83 Thread: pool-1-thread-3
Running: 69 Thread: pool-1-thread-1
Running: 19 Thread: pool-1-thread-2

# Threads: 5
Threads: [pool-1-thread-1, pool-1-thread-3, pool-1-thread-2, pool-1-thread-5, pool-1-thread-4]

We can see the the execution order is random, besides, the pool used the whole 5 threads to process the 100 tasks.

CACHED POOL

We can create an ExecuteService with a cached thread pool. In this case, we cannot control how many threads we have, the pool defines when creates or destroys a thread depending on the tasks load it has, which means, the pool grows or shrinks when it is necessary. A good use case for this pool is when you are not sure how many threads you need, to handle the load, but it is risky as the pool could grow unbounded. Usually if we have short live tasks, this pool is ideal.

  @Test
  public void cachedPool()
          throws InterruptedException {
    ExecutorService executor = Executors
            .newCachedThreadPool();

    submitTasks(executor);
  }

Running this test we got:

Running: 0 Thread: pool-1-thread-1
Running: 5 Thread: pool-1-thread-6
Running: 2 Thread: pool-1-thread-3
Running: 4 Thread: pool-1-thread-5
Running: 8 Thread: pool-1-thread-9
Running: 7 Thread: pool-1-thread-8
Running: 9 Thread: pool-1-thread-10
Running: 1 Thread: pool-1-thread-2
Running: 3 Thread: pool-1-thread-4
Running: 6 Thread: pool-1-thread-7
Running: 18 Thread: pool-1-thread-3

.....

Running: 75 Thread: pool-1-thread-18
Running: 12 Thread: pool-1-thread-10
Running: 82 Thread: pool-1-thread-4
Running: 83 Thread: pool-1-thread-7
Running: 81 Thread: pool-1-thread-2
Running: 80 Thread: pool-1-thread-5
Running: 79 Thread: pool-1-thread-8
Running: 76 Thread: pool-1-thread-15
Running: 97 Thread: pool-1-thread-18
Running: 94 Thread: pool-1-thread-19

# Threads: 19
Threads: [pool-1-thread-7, pool-1-thread-6, pool-1-thread-9, pool-1-thread-8, pool-1-thread-19, pool-1-thread-1, pool-1-thread-18, pool-1-thread-17, pool-1-thread-3, pool-1-thread-16, pool-1-thread-2, pool-1-thread-15, pool-1-thread-5, pool-1-thread-14, pool-1-thread-4, pool-1-thread-13, pool-1-thread-12, pool-1-thread-11, pool-1-thread-10]

We can see the the execution order is random, besides, the pool used 19 threads to process the 100 concurrent tasks. Take into account that the pool grow and shrink over time.

There is another kind of pool in Java, a Scheduled Pool.

Scheduled Thread Pools in Java: ScheduledExecutorService

Now, there is another kind of thread pool in Java that helps us with scheduled executions: The ScheduledExecutorService interface. This interface helps to schedule tasks that runs on the future.

Let’s see some Java code:

  @Test
  public void singleThreadPool()
          throws InterruptedException {
    ScheduledExecutorService executor = Executors
            .newSingleThreadScheduledExecutor();
//    ScheduledExecutorService executor = Executors
//            .newScheduledThreadPool(5);

    executor.schedule(() -> System.out.println(
            "Task 1 run once on " + LocalDateTime
                    .now()), 2, TimeUnit.SECONDS);

    executor.scheduleAtFixedRate(() -> System.out.println(
            "Task 2 run each 2 seconds " + LocalDateTime
                    .now()), 2, 2, TimeUnit.SECONDS);

    executor.scheduleWithFixedDelay(() -> System.out.println(
            "Task 3 run after 2 seconds finishing the previous one " + LocalDateTime
                    .now()), 2, 2, TimeUnit.SECONDS);

    Thread.sleep(10000);
  }

We see there two kind of scheduled pools: single thread and pool, the first one is equal as the Thread Pool we saw before, and the second one is the same as the Fixed Thread Pool.

Now, we have three methods to schedule tasks: schedule, scheduleAtFixedRate and scheduleWithFixedDelay:

• schedule: Schedule a task to be executed after X time of now, in the example, we have 2 seconds after now.
• scheduleAtFixedRate: Schedule a task to be executed after a delay plus a time, from now. In the example we delay 2 seconds and the task starts executing 2 seconds after.
• scheduleWIthFixedDelay: Schedule a tasks to be executed after a delay plus a delay after the last task finished. In the example, we schedule a task with 2 seconds delay and 2 seconds delay after the previous task finished.

The output of that test looks like the following:

Task 1 run once on 2021-04-01T19:08:23.151313
Task 2 run each 2 seconds 2021-04-01T19:08:23.153469
Task 3 run after 2 seconds finishing the previous one 2021-04-01T19:08:23.153755
Task 2 run each 2 seconds 2021-04-01T19:08:25.146104
Task 3 run after 2 seconds finishing the previous one 2021-04-01T19:08:25.154247
Task 2 run each 2 seconds 2021-04-01T19:08:27.142733
Task 3 run after 2 seconds finishing the previous one 2021-04-01T19:08:27.154686
Task 2 run each 2 seconds 2021-04-01T19:08:29.142782
Task 3 run after 2 seconds finishing the previous one 2021-04-01T19:08:29.155742
Task 2 run each 2 seconds 2021-04-01T19:08:31.142751
Task 3 run after 2 seconds finishing the previous one 2021-04-01T19:08:31.156637

Final Thought

Thread pools are a pretty good solution for the Java thread model regarding scalability, however, there are trade off to make here when we need to use them.

Any Java webserver uses a thread pool inside to handle the incoming requests, so, if you think you haven’t used thread pools, well, you might be wrong. Be aware of how your tools and frameworks works inside is important for troubleshooting and innovation.

In the following post, we will talk about futures, and how they helps us to grab results we expect to get in the future.

If you like this post and are interested in hearing more about my journey as a Software Engineer, you can follow me on Twitter and travel together.