Comp:clusters - Revision history

Gerritg: /* We have a new and automated system to reseverve cluster nodes */

2021-02-10T21:32:25Z

We have a new and automated system to reseverve cluster nodes

← Older revision		Revision as of 21:32, 10 February 2021
Line 17:		Line 17:
	The Slack/Yoink system is no longer in use.		The Slack/Yoink system is no longer in use.

	~~This is done on~~ Slack ~~with the~~ yoink-app. Before you submit a job, check under #cluster-usage which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.		== The Slack yoink-app that we no longer use ==
			Before you submit a job, check under #cluster-usage which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.
	To claim a resource:		To claim a resource:
	/yoink cn** [message] [lock duration]		/yoink cn** [message] [lock duration]

Gerritg: Add link to automated cluster reservation system

2021-02-10T21:31:07Z

Add link to automated cluster reservation system

← Older revision		Revision as of 21:31, 10 February 2021
Line 10:		Line 10:
	For other jobs you would like slurm to decide which clusternode will be best to use. In this case you do not use the -w flag. Here you need to make sure that all clusternodes have access to your files and you have to start your script from a subdirectory of your home directory.		For other jobs you would like slurm to decide which clusternode will be best to use. In this case you do not use the -w flag. Here you need to make sure that all clusternodes have access to your files and you have to start your script from a subdirectory of your home directory.

	Since we have many cluster with a smaller number of cores, it is sometimes hard for jobs that require many processors from the same node to start. This can be if many single-core jobs are running on many different nodes. For this reason it is possible to claim a clusternode for some time. This is done on Slack with the yoink-app. Before you submit a job, check under #cluster-usage which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.		Since we have many cluster with a smaller number of cores, it is sometimes hard for jobs that require many processors from the same node to start. This can be if many single-core jobs are running on many different nodes. For this reason it is possible to claim a clusternode for some time.

			== We have a new and automated system to reseverve cluster nodes ==
			So follow this link: [https://wiki.theochem.ru.nl/Comp:cluster_reservation cluster node reservation system]

			The Slack/Yoink system is no longer in use.

			This is done on Slack with the yoink-app. Before you submit a job, check under #cluster-usage which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.
	To claim a resource:		To claim a resource:
	/yoink cn** [message] [lock duration]		/yoink cn** [message] [lock duration]

Hcuppen at 15:11, 2 November 2020

2020-11-02T15:11:34Z

← Older revision		Revision as of 15:11, 2 November 2020
Line 10:		Line 10:
	For other jobs you would like slurm to decide which clusternode will be best to use. In this case you do not use the -w flag. Here you need to make sure that all clusternodes have access to your files and you have to start your script from a subdirectory of your home directory.		For other jobs you would like slurm to decide which clusternode will be best to use. In this case you do not use the -w flag. Here you need to make sure that all clusternodes have access to your files and you have to start your script from a subdirectory of your home directory.

	~~Look also~~ on ~~slack~~ #cluster-usage ~~to see~~ which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.		Since we have many cluster with a smaller number of cores, it is sometimes hard for jobs that require many processors from the same node to start. This can be if many single-core jobs are running on many different nodes. For this reason it is possible to claim a clusternode for some time. This is done on Slack with the yoink-app. Before you submit a job, check under #cluster-usage which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.
	To claim a resource:		To claim a resource:
	/yoink cn** [message] [lock duration]		/yoink cn** [message] [lock duration]

Hcuppen: Created page with "[https://cricket.science.ru.nl/ List with clustermachines (Only works with vpn)] Runs on this machine should go via a queue system. First read [https://wiki.cncz.science.ru.n..."

2020-11-02T15:03:18Z

Created page with "[https://cricket.science.ru.nl/ List with clustermachines (Only works with vpn)] Runs on this machine should go via a queue system. First read [https://wiki.cncz.science.ru.n..."

New page

[https://cricket.science.ru.nl/ List with clustermachines (Only works with vpn)]

Runs on this machine should go via a queue system. First read [https://wiki.cncz.science.ru.nl/Slurm the wiki-page of CNCZ] about its usage.

Our partition is
thchem.
How to send things to a cluster depends a bit on how data heavy your job is. For simulations with large trajectory files (lammps, vasp) you typically send them to a specific cluster node (with the flag -w). Here also the restart files/input data can be rather large.
The simulations are then run from the /scratch or /scratch2 partition. These partition are local on the cluster and are only locally accessible from the specific node. This makes them faster in writing and less dependable on the network.
Always make a directory with your login name from which you will work (with subdirectories). Look [https://www.theochem.ru.nl/DMP.html here] for more information to use scratch.
For other jobs you would like slurm to decide which clusternode will be best to use. In this case you do not use the -w flag. Here you need to make sure that all clusternodes have access to your files and you have to start your script from a subdirectory of your home directory.

Look also on slack #cluster-usage to see which nodes are claimed or not. You can only communicate with yoinkbot in the #cluster-usage channel and by direct message.
To claim a resource:
/yoink cn** [message] [lock duration]
To release a resource:
/release cn**
With a direct message to yoinkbot "resources list" you get a list with all claimed resources or by typing
"/dm @yoinkbot resources list"
in #cluster-usage

The first time a cluster node is claimed it will ask you whether you want to create this. Just confirm.
For electrostructure calculations, the output is typically small (can be in home), but the program needs to store large intermediate files (put on scratch, make sure they are deleted at the end of simulation).

Ask for example slurm scripts to someone who run similar jobs.