Access Dispynode through SSH-Tunnel/Forwarding

[uchendu-zhaw]

Target
Run a computation on a dispynode that is only accessible through SSH-tunnel.


Setup
My PC is in the same network as the dispynode server. The server is only accessible over SSH, meaning no other port can be used. The following schema describes roughly, what I try to achieve:


I do setup the port forwarding on the client machine, using the command:

Code: Select all

ssh -N -L 61591:warp03:61591 warp03 -v

This connection works kind of as expected. If I start on the server dispynode.py on the default port, I establish a connection from my PC using netcat.
Command on my PC:

Code: Select all

date | netcat localhost 61591

Output in dispynode on server (where 160.x.x.x is the network internal IP address of warp03):

Code: Select all

2021-04-29 08:53:46 dispynode - Ignoring invalid request from 160.x.x.x:47574

This is the output when I start my dispynode:

Code: Select all

2021-04-29 08:53:07 dispynode - version: 4.14.0 (Python 3.6.9), PID: 12739
2021-04-29 08:53:07 dispynode - Files will be saved under "/tmp/dispy/node"
2021-04-29 08:53:07 pycos - version 4.12.0 (Python 3.6.9) with epoll I/O notifier
2021-04-29 08:53:07 dispynode - "warp03" serving 52 cpus
2021-04-29 08:53:07 dispynode - TCP server at 160.x.x.x:61591

Problem
So far so good, but the compute request of the dispy client does reach/is not executed on the server. Meaning that I can see the forwarding of a request to the server in my SSH-tunnel, but in the log of the dispynode server nothing appears...

Configuration of my cluster in the program of my client:

Code: Select all

cluster = dispy.JobCluster(compute, nodes=["127.0.0.1"])

Output log of the SSH-tunnel:

Code: Select all

debug1: Connection to port 61591 forwarding to warp03 port 61591 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 61591 for warp03 port 61591, connect from 127.0.0.1 port 55198 to 127.0.0.1 port 61591, nchannels 3

What I've tried further
In the documentation of dispy it's stated that (https://dispy.org/dispy.html#ssh-port-forwarding):

If nodes are on remote network, nodes may not be able to communicate with client and NAT/Firewall Forwarding may not be possible. In such cases, SSH can be used for port forwarding.

To use this with JobCluster, client port (default is 61590) should be forwarded from each node with ssh -R 61590:127.0.0.1:61590 node, and then parameters ext_host=127.0.0.1 should be set to JobCluster.

I'm not 100% sure to understand it correctly, but another SSH-tunnel might be required to enable the communication between the node and the client. As a consequence, I created another tunnel by running this command on my PC to forward all requests from warp03:61590 to my local client without observing another behavior when running the example program:

Code: Select all

ssh -N -R 61590:127.0.0.1:61590 warp03 -v

[Giri]

In your case, it looke like both ports must be forwarded:

Code: Select all

ssh -R 61590:localhost:61590 -L 61591:localhost:61591 warp03

Then on warp03, start dispynode with 'dispynode -i 127.0.0.1' (this will give a warning about lookpback address but it is not a problem in this use case) and create cluster with

Code: Select all

cluster = dispy.JobCluster(compute, host="127.0.0.1", nodes=["127.0.0.1"])

[uchendu-zhaw]

Yes, that's it! Thanks a lot, Giri!

Just for clarification, this is the final setup of the forwarded ports: