Jenkins Master-Slave setup and configuration with screenshots

Topic Tools

Howdy friends ….

As promised in our Jenkins Installation & Configuration article, here we are with the configuration of Jenkins Master-Slave setup.

We have already learnt about Jenkins Installation and basic Configuration along with setting up our first project and running a build for the same.

Jenkins supports the master-slaves architecture, i.e. many slaves works for a master. It is also known as Jenkins Distributed Builds.

Jenkins_Master_Slaves

A Jenkins master comes with the basic installation of Jenkins and in this configuration the master handles all tasks for your build system.

Now, someone might thinks whether they really need this Jenkins slave or they can run their project without it ?

True, its not mandatory to have Jenkins client, Your master Jenkins can also create the build (what we did in our last article)

But, that will sustain just for small projects where they have very less jobs (say 5-10 job in a day and that too not very frequent)

When, you are working with numbers of projects and wants to run multiple jobs – You need to configure slaves/aagents to serve this purpose.

Jenkins slaves/agents are nothing just a small Java “Client” processes that connects to the “Master” Jenkins instance over the Java Network Launch Protocol (JNLP).

Jenkins Master role is to schedule the build jobs, assign slaves/agents and send builds to the slaves for the actual execution.

It also monitor the slaves (taking them online and offline as and when required), getting back the response of the build results from slaves then showing the build results on console.

All of the job results are collated on the master node for easy viewing.

With this approach of Jenkins master-slaves, the actual workload of building projects are delegated to multiple “slave” nodes, which allows to run numerous projects and their jobs.

It will also allow you to run jobs on different environments, like Linux, Windows, MacOS, etc (as shown in above image).

You might need to run same test case on different environments in parallel, that is where this distributed approach helps you to achieve the desired results quickly.

Once you configures few Jenkins slaves/agents, you might remove the executors on the Jenkins master in order to free up Jenkins master resources, but this isn’t necessary.

You can install and configure several Jenkins Slaves/Agents on 1 machine considering its input/output, cpu, disk & memory states.

Having multiple slaves will allow you to work more quickly without wasting time on waiting for build executors.

So, lets proceed with setting up a slave/agent/client for our Jenkins Master.

First, check whether you have Java installed or not. (If not then install it in similar fashion how we have installed during Jenkins installation)

Go to Jenkins dashboard -> Manage Jenkins -> Manage Nodes.

jenkins_manage_plugins_page

jenkins_node_configuration_step0

Name the node and click on OK.

* Initially you will get only one option “Permanent Agent” or “Dumb Slave“(in earlier jenkins versions), but once you have one slave then you will get another option to copy the node from another node which named as “Copy Existing Node

jenkins_node_configuration_step11

You will get the node configuration page where you need to enter required details. Not all the details are mandatory but it would be good practice to enter as much information as you can so that another team member can get all the idea without troubling you.

jenkins_node_configuration_step55

Lets go through with all the parameters in the node configuration page as displayed above :

Name : Name of the Slave which should be unique.

Description : Optional , Description for this slave, but as I said earlier, it would be really helpful for other team members

# of executors : The maximum number of concurrent builds that Jenkins may perform on this agent. I have used just 1 executor for testing purpose, but a good value to start with would be the number of CPU cores on the machine. You can check the server stat and then define the number of executors.

Remote root directory : An agent needs to have a directory dedicated to Jenkins. Specify the path to this directory on the agent. It is best to use an absolute path, such as /var/jenkins/node1 or c:\jenkins\node1. This should be a path local to the agent machine. There is no need for this path to be visible from the master.

Labels : Labels (or tags) are used to group multiple agents into one logical group. Multiple labels must be separated by a space. For example, linux docker would assign two labels to the agent: linux and docker.

Usage : Controls how Jenkins schedules builds on this node. Utilize this node as much as possible- This is the default and normal setting. In this mode, Jenkins uses this node freely

Launch method : It Controls how Jenkins starts this agent.

– Launch agent via Java Web Start: It allows slave to be launched using Java Web Start. In this case, a JNLP file must be opened on the agent machine, which will establish a TCP connection to the Jenkins master. This means that the agent need not be reachable from the master; the agent just needs to be able to reach the master. If you have enabled security via the Configure Global Security page, you can customize the port on which the Jenkins master will listen for incoming JNLP agent connections.
By default, the JNLP agent will launch a GUI, but its also possible to run a JNLP agent without a GUI, e.g. as a Window service.

– Launch agent via execution of command on the master: It starts slave by having Jenkins execute a command from the master. Use this when the master is capable of remotely executing a process on another machine, e.g. via SSH or RSH.

– Launch slave agents via SSH: It starts a slave by sending commands over a secure SSH connection. The slave needs to be reachable from the master, and you will have to supply an account that can log in on the target machine. No root privileges are required. This is the one which i am using for my slave configuration.

— Host Key Verification Strategy : Controls how Jenkins verifies the SSH key presented by the remote host whilst connecting.

— Known Host file verification strategy : Checks the known_hosts file (~/.ssh/known_hosts) for the user Jenkins is executing under, to see if an entry exists that matches the current connection.

{ If you are getting a SSH Host Key Verification error as below:

It could be a problem with SSH lib used by Jenkins which does not support newer ciphers like ecdsa-sha2-nistp256. Just delete the known_hosts entry and create a new one through below command

It will solve your problem.  }

— Mannually provided key verification strategy : Checks the key provided by the remote host matches the key set by the user who configured this connection.

— Mannually trusted key verification strategy : Require a user with Computer.CONFIGURE permission to authorise the key presented during the first connection to this host before the connection will be allowed to be established.

— Non Verifying verification strategy : Does not perform any verification of the SSH key presented by the remote host, allowing all connections regardless of the key they present. Its not advisable to select as it may open the path for attackers.

– Let Jenkins control this Windows slave as a Windows service: It starts a Windows slave by a remote management facility built into Windows. Suitable for managing Windows slaves. Slaves need to be IP reachable from the master.

Availability : Controls when Jenkins starts and stops this agent.

Keep this slave on-line as much as possible : This one is the default and normal setting. In this mode, Jenkins tries to keep the slave on-line as much as possible. If Jenkins can start the slave without user assistance, it will periodically attempt to restart the slave if it is unavailable. Jenkins will not take the slave off-line.

Take this slave on-line when in demand and off-line when idle : In this mode, if Jenkins can launch the slave without user assistance, it will periodically attempt to launch the slave while there are unexecuted jobs else the slave will be taken off-line by Jenkins

You can also define the node properties, like environment variables, such as PATH, JAVA_HOME, etc and Tools locations such as, ANT, GIT, etc.

jenkins_node_configuration_step4

Environment variables defined here will be made available to every build executed by this agent, and will override any environment variables that have the same Name as those defined on the Configure System page.

For Tool Locations, you can specify the location of certain tools on this node, overriding the global configuration (You may prefer to use automatic tool installers instead, removing the need to configure each node separately)

Once you are done click on SAVE and that’s it. Your new Jenkins Slave is up and running and ready to take the builds

jenkins_node_status01

Now, lets go ahead and run our first build on this slave.

As I have mentioned and you might have noted that Jenkins master do have their build executors and until we turn them off, Jenkins can use those executors to perform builds based on the CPU stats, memory and other factors. So to avoid this issue we need to do some changes in our project configuration to force the project’s job to run on newly created Jenkins Slave.

This technique is also helpful when you have some jobs to be run on different platforms, with specific configurations, etc (like we discussed in the beginning of this article)

So, to configure the project specific to this new node,

Open the Project and click on Configure

jenkins_node_run1

Find the check-box in General section “Restrict where this project can be run” and click on it

jenkins_node_run2

Just type the Label name of your slave, in this case its “node1”

jenkins_node_run3

And then Save the changes.

Go back again to Your Project and click on “Build with Parameters

jenkins_node_run1

Choose the appropriate parameter(s) [Based on your project configurations] and click on Build

jenkins_node_run4

Then click on Build number and check the Console Output, where you will find that build is running on newly created node and is successful

jenkins_node_run5

You can also check the Node statistics to get all the idea about the newly created node/slave/agent.

jenkins_node_run6

You can also define more than one labels (i.e. more than one node to the project) which will allow you to run the build in parallel and Jenkins will choose the faster slave for your build. To do that,

Go back to Project Configuration and find the “Restrict where this project can be run” option. Click on Help for feature: This project is parameterised button to know all the expression details

jenkins_node_configuration_step7

You can use || symbol to tell Jenkins to use any of the slave, like below

jenkins_node_configuration_step8

Once done, Save your changes and again go back to Your Project and click on “Build with Parameters“. And this time you will find the job ran on master build executor since we have given a choice to Jenkins to choose any build executor of master or slave (node1)

jenkins_run2

Congratulations !! You have successfully added the new Jenkins Slave/Agent/Node and configured project accordingly.

In today’s article we have covered:

  • Need of Jenkins Slave/Agent
  • Concept of Distributed Build
  • Jenkins Slave/Agent configuration
  • Issue with SSH Host Key Verification and its resolution
  • Jenkins Project changes to run job on particular node
  • Build on Jenkins Agent

Hope you enjoyed our article and got more idea about Jenkins.

We will come with some new topic, till then stay tuned ….

18 comments… add one

  • Amit

    I was actually waiting for this master-slave setup after reading your previous article on Jenkins installation. Thanks a bunch !!

  • Henry

    Awesome. You have described it very well, specially the issue with ssh host key verification. Nice work

  • Sameer

    Nice one Ravi !!

  • Mahesh

    Thanks for this wonderful article, it has actually helped me to configure my jenkins node on linux machine but I am facing error with windows jenkins node, can you please write on that as well ?

  • Tom

    You cover the node configuration well but I quickly read the article 3 times and still can’t see you mention how to install the client on the remote machine?

  • Ethan

    Please share for Windows slave configuration – thanks

  • Mridul

    Thanks very much. This is an excellent article providing insight into the Master-Slave concept of Jenkins.

  • Namdev

    Hello,

    I have installed Jenkins server on windows server and I have repos on GIT Server.

    I want to deploy php and some file on windows server.

    Can anyone help me how to add windows client in jenkins and deploy code on windows. which plugins is suitable and also share if any links.

  • Su Le

    Hi Ravi,
    Thanks for this post, I had a common case, the jenkins server is installed on Mac and we have ASP.NET project, so we need to build ASP.NET project via dotnet build tool on another Windows machine (a Node), so how I can execute dotnet build tool or cmd on that Windows Node? Thank you.

    • Hi Su,
      Thanks for your query.
      You can refer the article http://scmquest.com/jenkins-windows-slave-configuration-with-screenshots/ for configuring windows slave
      Once it is configured you can tied the new windows node to your ASP.Net project and use msbuild, etc for building the project.
      In order to use MSBuild, you need to install the plugin
      – Click the “Configure System” through “Manage Jenkins”
      – Search for “MSBuild”
      – Click on “MSBuild installations”
      – then click “Add MSBuild”
      – Make sure to give the fully qualified path to msbuild.exe in path field

      Once that is done:
      -Navigate to your “Project”
      -Select “Configure”
      -Go to “Add build step”
      -Select “Build a Visual Studio project or solution using MSBuild” option
      -For other commands you can use : Select “Execute Windows batch command”

      And you are good to go
      Hope it helps and do let me know in case of any issue.

  • Elvis T

    Nice work

  • Rinku

    Whats the default value of #executor if executor is left empty while configuration?

    • Hi Rinku,
      Thanks for your query.
      The default value of # of Executors for the slave is : 1, when you don’t edit the value.
      You can also refer the documentation for this field by clicking on question mark info symbol found next to field
      “Agents must have at least one executor. To temporarily prevent any builds from being executed on an agent, use the Mark this node temporarily offline button on the agent’s page.
      This does not apply to the Jenkins master — setting the number of executors to zero will prevent any builds from being executed on the master.”

      And if you edit this value to 0 for Slave/Agent Node – It will throw the error as “Invalid agent configuration for ‘agentname’ Invalid # of executors.”

      As I highlighted in the article, a good value to start with would be the number of CPU cores on the machine. You can check the server stat and then define the number of executors.
      Hope this helps ..

  • Milan

    Nicely put. Appreciate it!

  • Abhay Tiwari

    We need to deploy war file in a centralized system, in KIOSK machine we are deploying war file in each system, how to resolve this issue , we are looking for a deployment centrally and all nodes in same network can access that.

    • Hi Abhay,
      Thanks for posting your query.
      Please can you provide more details:
      – What is the input (tools, technology, build tool, etc)
      – What is an expected outcome ( Build output, App/Web Servers, Deployment type, Clusters/ Node statistic, etc)
      – Your Network Topology
      – Issue you are facing with debug output.

      Will then go through with your issues and try to resolve/suggest the solution

Leave a Comment