I have setup a seperate schduler. For now every first(, third, fith, ...) run fails and every second(, fourth, sixth, ...) run works. Dispy version ist pips.
For testing i use a modified example from the documentation:
Code: Select all
#!/usr/bin/python3
def compute(n):
import time, socket
time.sleep(n)
host = socket.gethostname()
return (host, n)
if __name__ == '__main__':
import dispy, random
import time
cluster = dispy.SharedJobCluster(compute, host='a.b.c.1', client_port=0, scheduler_host='a.b.c.1')
time.sleep(5)
jobs = []
for i in range(2000):
# schedule execution of 'compute' on a node (running 'dispynode')
# with a parameter (random number in this case)
job = cluster.submit(1) #random.randint(5,20))
# job.id = i # optionally associate an ID to job (if needed later)
jobs.append(job)
cluster.wait() # waits for all scheduled jobs to finish
for job in jobs:
host, n = job() # waits for job to finish and returns results
print('%s executed job %s at %s with %s' % (host, job.id, job.start_time, n))
# other fields of 'job' that may be useful:
# print(job.stdout, job.stderr, job.exception, job.ip_addr, job.start_time, job.end_time)
cluster.print_status()
cluster.close()
Code: Select all
2021-11-24 09:40:46 dispy - Creating job for "(5,)", "{}" failed with "Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/dispy/__init__.py", line 3330, in submit_job_id_node
_job.uid = deserialize(msg)
File "/usr/local/lib/python3.9/dist-packages/pycos/__init__.py", line 85, in deserialize
return pickle.loads(pkl)
EOFError: Ran out of input
"
Traceback (most recent call last):
File "/home/beekeeper/data/./test3.py", line 24, in <module>
host, n = job() # waits for job to finish and returns results
TypeError: 'NoneType' object is not callable
the scheduler setup:
Code: Select all
dispyscheduler.py -d --httpd --host=a.b.c.1 --node_secret 123456 --fair_cluster_schedulerCode: Select all
dispynode.py --clean --scheduler_host=a.b.c.1 --secret=123456 --admin_secret 654321
)(or I am the only one trying to use dispyscheduler and now all the bugs come out 
).