TCP/IP Discovery

❗️
This is a legacy Apache Ignite documentation
The new documentation is hosted here: https://ignite.apache.org/docs/latest/

Overview

In Ignite, nodes can discover each other by using DiscoverySpi. Ignite provides TcpDiscoverySpi as a default implementation of DiscoverySpi that uses TCP/IP for node discovery. Discovery SPI can be configured for Multicast and Static IP based node discovery.

Multicast IP Finder

TcpDiscoveryMulticastIpFinder uses Multicast to discover other nodes in the grid and is the default IP finder. You should not have to specify it unless you plan to override default settings. Here is an example of how to configure this finder via a Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
ipFinder.setMulticastGroup("228.10.10.157");
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Static IP Finder

For cases when Multicast is disabled, TcpDiscoveryVmIpFinder should be used with pre-configured list of IP addresses.

You are only required to provide at least one IP address of a remote node, but usually it is advisable to provide 2 or 3 addresses of grid nodes that you plan to start at some point of time in the future. Once a connection to any of the provided IP addresses is established, Ignite will automatically discover all other grid nodes.

📘
Instead of specifying addresses in configuration, you can specify them in the IGNITE_TCP_DISCOVERY_ADDRESSES environment variable or in the system property with the same name. Addresses should be comma separated and may contain an optional port range.

🚧
By default the TcpDiscoveryVmIpFinder is used in non-shared mode. If you plan to start a server node then in this mode the list of IP addresses should contain an address of the local node as well. It will let the node not to wait while other nodes join the cluster but rather become the first cluster node and operate normally.

Here is an example of how to configure this finder via a Spring XML file or programmatically from Java:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
          <property name="addresses">
            <list>
              <!-- 
              Explicitly specifying address of a local node to let it start and
              operate normally even if there is no more nodes in the cluster. 
              You can also optionally specify an individual port or port range.
              -->
              <value>1.2.3.4</value>
              
              <!-- 
              IP Address and optional port range of a remote node.
              You can also optionally specify an individual port.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>

TcpDiscoverySpi spi = new TcpDiscoverySpi();
 
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
 
// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

# The configuration should use TcpDiscoveryVmIpFinder without addresses specified:

IGNITE_TCP_DISCOVERY_ADDRESSES=1.2.3.4,1.2.3.5:47500..47509 bin/ignite.sh -v config/default-config.xml

Multicast and Static IP Finder

You can use both, Multicast and Static IP based discovery together. In this case, in addition to addresses received via multicast, if any, TcpDiscoveryMulticastIpFinder can also work with pre-configured list of static IP addresses, just like Static IP-Based Discovery described above. Here is an example of how to configure Multicast IP finder with static IP addresses:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
          <property name="multicastGroup" value="228.10.10.157"/>
           
          <!-- list of static IP addresses-->
          <property name="addresses">
            <list>
              <value>1.2.3.4</value>
              
              <!-- 
                  IP Address and optional port range.
                  You can also optionally specify an individual port.
              -->
              <value>1.2.3.5:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
</bean>

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
 
// Set Multicast group.
ipFinder.setMulticastGroup("228.10.10.157");

// Set initial IP addresses.
// Note that you can optionally specify a port or a port range.
ipFinder.setAddresses(Arrays.asList("1.2.3.4", "1.2.3.5:47500..47509"));
 
spi.setIpFinder(ipFinder);
 
IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Isolated Ignite Clusters on Same Set of Machines

Ignite allows you to start two isolated clusters on the same set of machines. This can be done if nodes from different clusters use non intersecting local port ranges for TcpDiscoverySpi and TcpCommunicationSpi.

Let's say that you need to start two isolated clusters on a single machine for testing purposes.
For the nodes from the first cluster, you should use the following TcpDiscoverySpi and TcpCommunicationSpi configurations:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <!--
    Explicitly configure TCP discovery SPI to provide list of 
    initial nodes from the first cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="48500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of nodes from 
                            the first cluster.
                            127.0.0.1 can be replaced with actual IP addresses 
                            or host names. Port range is optional.
                            -->
                            <value>127.0.0.1:48500..48520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local
    port number for the nodes from the first cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="48100"/>
        </bean>
    </property>
</bean>

IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the first cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(48500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the first cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:48500..48520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the first cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(48100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

For the nodes from the second cluster, the configuration can look like this:

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <!--
    Explicitly configure TCP discovery SPI to provide list of initial
    nodes from the second cluster.
    -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <!-- Initial local port to listen to. -->
            <property name="localPort" value="49500"/>

            <!-- Changing local port range. This is an optional action. -->
            <property name="localPortRange" value="20"/>

            <!-- Setting up IP finder for this cluster -->
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <!--
                            Addresses and port range of the nodes from the second                             cluster.
                            127.0.0.1 can be replaced with actual IP addresses or                             host names. Port range is optional.
                            -->
                            <value>127.0.0.1:49500..49520</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>

    <!--
    Explicitly configure TCP communication SPI changing local port number 
    for the nodes from the second cluster.
    -->
    <property name="communicationSpi">
        <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
            <property name="localPort" value="49100"/>
        </bean>
    </property>
</bean>

IgniteConfiguration cfg = new IgniteConfiguration();

// Explicitly configure TCP discovery SPI to provide list of initial nodes
// from the second cluster.
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();

// Initial local port to listen to.
discoverySpi.setLocalPort(49500);

// Changing local port range. This is an optional action.
discoverySpi.setLocalPortRange(20);

TcpDiscoveryVmIpFinder ipFinder=new TcpDiscoveryVmIpFinder();

// Addresses and port range of the nodes from the second cluster.
// 127.0.0.1 can be replaced with actual IP addresses or host names.
// The port range is optional.
ipFinder.setAddresses(Arrays.asList("127.0.0.1:49500..49520"));

// Overriding IP finder.
discoverySpi.setIpFinder(ipFinder);

// Explicitly configure TCP communication SPI by changing local port number for
// the nodes from the second cluster.
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();

commSpi.setLocalPort(49100);

// Overriding discovery SPI.
cfg.setDiscoverySpi(discoverySpi);

// Overriding communication SPI.
cfg.setCommunicationSpi(commSpi);

// Starting a node.
Ignition.start(cfg);

As you see from the configurations the difference between them is minor - only port numbers for SPIs and IP finder vary.

📘
If you want the nodes from different clusters to be able to look for each other using multicast protocol, replace TcpDiscoveryVmIpFinder with TcpDiscoveryMulticastIpFinder and set unique TcpDiscoveryMulticastIpFinder.multicastGroups in each configuration above.

🚧
Ignite Persistence Files Location
If the isolated clusters use Ignite persistence, then every cluster has to store its persistence files under different paths in the file system. Use the setStoragePath(...), setWalPath(...) and setWalArchivePath(...) methods of DataStorageConfiguration to redefine the paths for every individual cluster.

Apache jclouds IP Finder

Refer to Generic Cloud Configuration documentation.

Amazon S3 IP Finder

Refer to AWS S3 Configuration documentation.

Amazon ELB IP Finder

Refer to AWS ELB Configuration documentation.

Google Cloud Storage IP Finder

Refer to Google Cloud Configuration documentation.

JDBC Based IP Finder

You can have your database be a common shared storage of initial IP addresses. With this IP finder, nodes will write their IP addresses to a database on startup. This is done via TcpDiscoveryJdbcIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.jdbc.TcpDiscoveryJdbcIpFinder">
          <property name="dataSource" ref="ds"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

<!-- Configured data source instance. -->
<bean id="ds" class="some.Datasource">
  ...
</bean>

TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configure your DataSource.
DataSource someDs = MySampleDataSource(...);

TcpDiscoveryJdbcIpFinder ipFinder = new TcpDiscoveryJdbcIpFinder();

ipFinder.setDataSource(someDs);

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Shared File System IP Finder

A shared file system can be used as a storage for nodes' IP addresses. The nodes will write their IP addresses to the file system on startup. This behavior is supported by TcpDiscoverySharedFsIpFinder.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
  ...
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.sharedfs.TcpDiscoverySharedFsIpFinder">
          <property name="path" value="/var/ignite/addresses"/>
        </bean>
      </property>
    </bean>
  </property>
</bean>

// Configuring discovery SPI.
TcpDiscoverySpi spi = new TcpDiscoverySpi();

// Configuring IP finder.
TcpDiscoverySharedFsIpFinder ipFinder = new TcpDiscoverySharedFsIpFinder();

ipFinder.setPath("/var/ignite/addresses");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Kubernetes IP Finder

Refer to Kubernetes IP Finder documentation.

ZooKeeper IP Finder

If you're using ZooKeeper to coordinate your distributed environment, you can utilize it as:

Foundation of the overall discovery component (see ZooKeeper Discovery for more details).
Use it as an IP finder. This section explains this use case.

To set up ZooKeeper IP finder use TcpDiscoveryZookeeperIpFinder (note that ignite-zookeeper module has to be enabled).

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    ...
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
                    <property name="zkConnectionString" value="127.0.0.1:2181"/>
                </bean>
            </property>
        </bean>
    </property>
</bean>

TcpDiscoverySpi spi = new TcpDiscoverySpi();

TcpDiscoveryZookeeperIpFinder ipFinder = new TcpDiscoveryZookeeperIpFinder();

// Specify ZooKeeper connection string.
ipFinder.setZkConnectionString("127.0.0.1:2181");

spi.setIpFinder(ipFinder);

IgniteConfiguration cfg = new IgniteConfiguration();
 
// Override default discovery SPI.
cfg.setDiscoverySpi(spi);
 
// Start Ignite node.
Ignition.start(cfg);

Failure Detection Timeout

Failure detection timeout is used to determine how long a cluster node should wait before considering a remote connection, with another node, failed.

Every node in the Ignite cluster is connected with another node, at the level of discovery SPI. NodeA sends heartbeats (and other system messages transferred over the cluster ring - discovery SPI) to nodeB, and if the latter doesn’t reply in failureDetectionTimeout, then nodeB will be kicked off the cluster.

This timeout is the easiest way to tune discovery SPI's failure detection feature depending on the network and hardware conditions of your environment.

🚧
The timeout automatically controls configuration parameters of TcpDiscoverySpi, such as socket timeout, message acknowledgment timeout and others. If any of these parameters is set explicitly, then the failure timeout setting will be ignored.

The failure detection timeout is configured using IgniteConfiguration.setFailureDetectionTimeout(long) for Apache Ignite server nodes and IgniteConfiguration.setClientFailureDetectionTimeout(long) for client nodes. The default value is equal to 10 seconds for the server nodes and 30 seconds for the client nodes. This allows the discovery SPI to work reliably on most of the on-premise and containerized deployments. However, for stable low-latency networks, the parameter can be set to ~200 milliseconds in order to detect and react to failures more quickly.

Configuration

Below you can see the most frequently used TcpDiscoverySpi configuration parameters. Refer to TcpDiscoverySpi javadoc to see the full list of configuration options.

Setter Method	Description	Default
`setIpFinder(TcpDiscoveryIpFinder)`	IP finder that is used to share info about nodes IP addresses.	`TcpDiscoveryMulticastIpFinder` Some of the implementations that can be used: `TcpDiscoverySharedFsIpFinder` `TcpDiscoveryS3IpFinder` `TcpDiscoveryJdbcIpFinder` `TcpDiscoveryVmIpFinder`
`setLocalAddress(String)`	Sets local host IP address that discovery SPI uses.	If not provided, by default a first found non-loopback address will be used. If there is no non-loopback address available, then `java.net.InetAddress.getLocalHost()` will be used.
`setLocalPort(int)`	Port the SPI listens to.	47500
`setLocalPortRange(int)`	Local port range. Local node will try to bind on first available port starting from local port up until local port + local port range.	100
`setReconnectCount(int)`	Number of times node tries to (re)establish connection to another node.	2
`setNetworkTimeout(long)`	Sets maximum network timeout in milliseconds to use for network operations.	5000
`setSocketTimeout(long)`	Sets socket operations timeout. This timeout is used to limit connection time and write-to-socket time.	2000
`setAckTimeout(long)`	Sets timeout for receiving acknowledgement for sent message. If acknowledgement is not received within this timeout, sending is considered as failed and SPI tries to repeat message sending.	2000
`setJoinTimeout(long)`	Sets join timeout. If non-shared IP finder is used and node fails to connect to any address from IP finder, node keeps trying to join within this timeout. If all addresses are still unresponsive, exception is thrown and node startup fails. 0 means wait forever.	0
`setThreadPriority(int)`	Thread priority for threads started by SPI.	0
`setStatisticsPrintFrequency(int)`	Statistics print frequency in milliseconds. 0 indicates that no print is required. If value is greater than 0 and log is not quiet then stats are printed out with INFO level once a period. This may be very helpful for tracing topology problems.	true

TCP/IP Discovery

❗️
This is a legacy Apache Ignite documentation

Overview

Multicast IP Finder

Static IP Finder

📘
Instead of specifying addresses in configuration, you can specify them in the `IGNITE_TCP_DISCOVERY_ADDRESSES` environment variable or in the system property with the same name. Addresses should be comma separated and may contain an optional port range.

Multicast and Static IP Finder

Isolated Ignite Clusters on Same Set of Machines

📘
If you want the nodes from different clusters to be able to look for each other using multicast protocol, replace `TcpDiscoveryVmIpFinder` with `TcpDiscoveryMulticastIpFinder` and set unique `TcpDiscoveryMulticastIpFinder.multicastGroups` in each configuration above.

🚧
Ignite Persistence Files Location

Apache jclouds IP Finder

Amazon S3 IP Finder

Amazon ELB IP Finder

Google Cloud Storage IP Finder

JDBC Based IP Finder

Shared File System IP Finder

Kubernetes IP Finder

ZooKeeper IP Finder

Failure Detection Timeout

🚧

Configuration

❗️This is a legacy Apache Ignite documentation

Overview

Multicast IP Finder

Static IP Finder

📘Instead of specifying addresses in configuration, you can specify them in the IGNITE_TCP_DISCOVERY_ADDRESSES environment variable or in the system property with the same name. Addresses should be comma separated and may contain an optional port range.

Multicast and Static IP Finder

Isolated Ignite Clusters on Same Set of Machines

📘If you want the nodes from different clusters to be able to look for each other using multicast protocol, replace TcpDiscoveryVmIpFinder with TcpDiscoveryMulticastIpFinder and set unique TcpDiscoveryMulticastIpFinder.multicastGroups in each configuration above.

🚧Ignite Persistence Files Location

Apache jclouds IP Finder

Amazon S3 IP Finder

Amazon ELB IP Finder

Google Cloud Storage IP Finder

JDBC Based IP Finder

Shared File System IP Finder

Kubernetes IP Finder

ZooKeeper IP Finder

Failure Detection Timeout

🚧

Configuration

❗️
This is a legacy Apache Ignite documentation

📘
Instead of specifying addresses in configuration, you can specify them in the `IGNITE_TCP_DISCOVERY_ADDRESSES` environment variable or in the system property with the same name. Addresses should be comma separated and may contain an optional port range.

📘
If you want the nodes from different clusters to be able to look for each other using multicast protocol, replace `TcpDiscoveryVmIpFinder` with `TcpDiscoveryMulticastIpFinder` and set unique `TcpDiscoveryMulticastIpFinder.multicastGroups` in each configuration above.

🚧
Ignite Persistence Files Location