Access Dispynode through SSH-Tunnel/Forwarding
Posted: Thu Apr 29, 2021 7:37 am
Target
Run a computation on a dispynode that is only accessible through SSH-tunnel.
Setup
My PC is in the same network as the dispynode server. The server is only accessible over SSH, meaning no other port can be used. The following schema describes roughly, what I try to achieve:
I do setup the port forwarding on the client machine, using the command:
This connection works kind of as expected. If I start on the server dispynode.py on the default port, I establish a connection from my PC using netcat.
Command on my PC:
Output in dispynode on server (where 160.x.x.x is the network internal IP address of warp03):
This is the output when I start my dispynode:
Problem
So far so good, but the compute request of the dispy client does reach/is not executed on the server. Meaning that I can see the forwarding of a request to the server in my SSH-tunnel, but in the log of the dispynode server nothing appears...
Configuration of my cluster in the program of my client:
Output log of the SSH-tunnel:
What I've tried further
In the documentation of dispy it's stated that (https://dispy.org/dispy.html#ssh-port-forwarding):
Run a computation on a dispynode that is only accessible through SSH-tunnel.
Setup
My PC is in the same network as the dispynode server. The server is only accessible over SSH, meaning no other port can be used. The following schema describes roughly, what I try to achieve:
I do setup the port forwarding on the client machine, using the command:
Code: Select all
ssh -N -L 61591:warp03:61591 warp03 -v
Command on my PC:
Code: Select all
date | netcat localhost 61591
Code: Select all
2021-04-29 08:53:46 dispynode - Ignoring invalid request from 160.x.x.x:47574
Code: Select all
2021-04-29 08:53:07 dispynode - version: 4.14.0 (Python 3.6.9), PID: 12739
2021-04-29 08:53:07 dispynode - Files will be saved under "/tmp/dispy/node"
2021-04-29 08:53:07 pycos - version 4.12.0 (Python 3.6.9) with epoll I/O notifier
2021-04-29 08:53:07 dispynode - "warp03" serving 52 cpus
2021-04-29 08:53:07 dispynode - TCP server at 160.x.x.x:61591
Problem
So far so good, but the compute request of the dispy client does reach/is not executed on the server. Meaning that I can see the forwarding of a request to the server in my SSH-tunnel, but in the log of the dispynode server nothing appears...
Configuration of my cluster in the program of my client:
Code: Select all
cluster = dispy.JobCluster(compute, nodes=["127.0.0.1"])Code: Select all
debug1: Connection to port 61591 forwarding to warp03 port 61591 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 61591 for warp03 port 61591, connect from 127.0.0.1 port 55198 to 127.0.0.1 port 61591, nchannels 3
What I've tried further
In the documentation of dispy it's stated that (https://dispy.org/dispy.html#ssh-port-forwarding):
I'm not 100% sure to understand it correctly, but another SSH-tunnel might be required to enable the communication between the node and the client. As a consequence, I created another tunnel by running this command on my PC to forward all requests from warp03:61590 to my local client without observing another behavior when running the example program:If nodes are on remote network, nodes may not be able to communicate with client and NAT/Firewall Forwarding may not be possible. In such cases, SSH can be used for port forwarding.
To use this with JobCluster, client port (default is 61590) should be forwarded from each node with ssh -R 61590:127.0.0.1:61590 node, and then parameters ext_host=127.0.0.1 should be set to JobCluster.
Code: Select all
ssh -N -R 61590:127.0.0.1:61590 warp03 -v
