Open OnDemand 3.0 install on Rocky Linux 8.7

This article is a collection of important notes with examples in addition to Open OnDemand documentation.

There are many changes between releases. This article shows an example of release 3.0.

I'm also using the Slurm batch queue system and "modules" (EasyBuild and Lmod). It is used in the configuration files below.

Visit Open OnDemand page to find the latest documentation:

https://openondemand.org

1. Only HTTPS TLS/SSL

Do not even start without SSL/TLS certificate.

You can go through documentation halfway and find that many things do not work without SSL/TLS certificate. So:

HTTP does not work. Link: https://github.com/OSC/ondemand/issues/1193
Self-signed certificate does not work. Error message: "...openid-configuration (SSL certificate problem: self signed certificate)"

2. Where is the log file?

Not every error event goes to the logs files. But these are 2 locations to start looking for errors:

/var/log/ondemand-nginx/
/var/log/httpd/

3. The configuration file does not exist

Examples in the documentation are mentioning different files or directories. Many of those files and directories do not exist. You need to create them. And when you search on the Internet. You can get links to old documentation for release 2.0 or 1.0 where files had different locations. This is the list of files and directories that I would love to have before I started with Open OnDemand:

/etc/ood/config/clusters.d
/etc/ood/config/clusters.d/my_ood.yml
/etc/ood/config/ood_portal.yml
/etc/ood/config/ondemand.d/
/etc/ood/config/ondemand.d/ondemand.yml
/etc/ood/config/apps/bc_desktop/
/etc/ood/config/apps/bc_desktop/ood_desktop.yml
/etc/ood/config/apps/dashboard/initializers/
/etc/ood/config/apps/dashboard/initializers/ood.rb
/var/www/ood/apps/sys/jupyter/template
/var/www/ood/apps/sys/bc_desktop/form.yml
/var/www/ood/apps/sys/bc_desktop/submit.yml.erb
/var/www/ood/public/
/etc/ood/config/nginx_stage.yml
/etc/pki/tls/private/ood_certificate.key
/etc/pki/tls/certs/ood_certificate.cer
/etc/pki/tls/certs/ood_certificate.pem
/var/www/ood/apps/sys/my_jupyter
/var/www/ood/apps/sys/my_matlab

4. Developers mode is disabled by default.

Users can develop new applications for Open OnDemand in their home directories. By creating this directory: /home//ondemand/dev

But it is disabled by default.

Edit this file: /etc/ood/config/nginx_stage.yml

# If you want to enable app development like in 1.3, where each user's home directory
# use this app_root block instead:
app_root:
  dev: '~%{owner}/%{portal}/dev/%{name}'
  usr: '/var/www/ood/apps/usr/%{owner}/gateway/%{name}'
  sys: '/var/www/ood/apps/sys/%{name}'

5. These are the important commands

/opt/ood/ood-portal-generator/sbin/update_ood_portal
systemctl restart httpd
systemctl restart ondemand-dex

6. You changed the config file but no changes applied?

Because the nginx server needs to restart.

Find “ps aux | grep nginx” and then kill the process.

Later I found that on the web interface, you can go to Help > Restart Web Server.

7. Firewall port to open

These 3 TCP ports: 80, 443, 5554

80 # just to forward to HTTPS
443 # https
5554 # Dex authentification.

8. LDAP

Are your cluster without LDAP? Then skip this section. Unfortunately, this is the most unpredictable section. Your LDAP server will be configured differently. Search for more examples on the Internet. This is what I added to the file /etc/ood/config/ood_portal.yml

. . .
dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: <server_name>:636
        insecureSkipVerify: false
        bindDN: uid=read_only,ou=systembruker,ou=system,dc=khmel,dc=org
        #bindDN: ou=users,dc=khmel,dc=org
        bindPW: 1a1a1a1cmcccccccccccccb2b2b2b
        userSearch:
          baseDN: ou=users,dc=khmel,dc=org
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: uid
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=groups,dc=khmel,dc=org
          filter: "(objectClass=myGroupMember)"
          userMatchers:
            - userAttr: DN
              groupAttr: myMemberOf
          nameAttr: cn

Commands after the change:

/opt/ood/ood-portal-generator/sbin/update_ood_portal
systemctl restart httpd
systemctl restart ondemand-dex

9. Shell web access does not work. Websocket issue.

Until you will add these lines to the file: /etc/ood/config/clusters.d/my_cluster.yml

---
v2:
  metadata:
    title: "my_cluster"
  login:
    host: "127.0.0.1"

This file needs more lines. See below.

10. Web interface cannot list active jobs:

Until you will add these lines to the file: /etc/ood/config/clusters.d/my_cluster.yml

---
v2:
  metadata:
    title: "my_cluster"
  login:
    host: "127.0.0.1"
  job:
    adapter: "slurm"

This file needs more lines. See below.

11. Desktop

Packages needed for compute nodes:

dnf install python3-websockify
dnf install turbovnc-3.0.3.x86_64.rpm
dnf group install "Xfce"

I tried MATE but got issues with the window's headers. I have not spent time debugging because I was planning to use the XFCE desktop.

But the desktop will not work until you update this file:

---
v2:
  metadata:
    title: "my_cluster"
  login:
    host: "127.0.0.1"
  job:
    adapter: "slurm"
  batch_connect:
    basic:
      script_wrapper: |
        module purge
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"
    vnc:
      script_wrapper: |
        module purge
        export PATH="/opt/TurboVNC/bin:$PATH"
        export WEBSOCKIFY_CMD="/usr/bin/websockify"
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"

12. How to install new applications.

You can start with examples from here: https://osc.github.io/ood-documentation/latest/install-ihpc-apps.html

For example Jupyter Notebooks:

cd /var/www/ood/apps/sys/
git clone https://github.com/OSC/bc_example_jupyter.git

Sometimes it is enough to edit only form.yml or form.yml.erb file.

For more complex changes 3 important files to edit:

/var/www/ood/apps/sys/bc_example_jupyter/form.yml
/var/www/ood/apps/sys/bc_example_jupyter/submit.yml.erb
/var/www/ood/apps/sys/bc_example_jupyter/template/script.sh.erb

file extention can be .yml or .yml.erb

13. Configuration file examples

File: /etc/ood/config/ood_portal.yml

---
servername: ood.khmel.org
ssl:
  - 'SSLCertificateFile "/etc/pki/tls/certs/ood.khmel.org.pem"'
  - 'SSLCertificateKeyFile "/etc/pki/tls/private/ood.khmel.org.key"'
  - 'SSLCertificateChainFile "/etc/pki/tls/certs/ood.khmel.org_int.cer"'
host_regex: '[^/]+'
node_uri: '/node'
rnode_uri: '/rnode'
dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: <server_name>:636
        insecureSkipVerify: false
        bindDN: uid=read_only,ou=systembruker,ou=system,dc=khmel,dc=org
        #bindDN: ou=users,dc=khmel,dc=org
        bindPW: 1a1a1a1cmcccccccccccccb2b2b2b
        userSearch:
          baseDN: ou=users,dc=khmel,dc=org
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: uid
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=groups,dc=khmel,dc=org
          filter: "(objectClass=myGroupMember)"
          userMatchers:
            - userAttr: DN
              groupAttr: myMemberOf
          nameAttr: cn

File /etc/ood/config/clusters.d/my_ood.yml

---
v2:
  metadata:
    title: "my_cluster"
  login:
    host: "127.0.0.1"
  job:
    adapter: "slurm"
  batch_connect:
    basic:
      script_wrapper: |
        module purge
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"
    vnc:
      script_wrapper: |
        module purge
        export PATH="/opt/TurboVNC/bin:$PATH"
        export WEBSOCKIFY_CMD="/usr/bin/websockify"
        %s
      set_host: "host=$(hostname -A | awk '{print $1}')"

File /etc/ood/config/ondemand.d/ondemand.yml

pinned_apps_group_by: subcategory
pinned_apps:
  - sys/bc_desktop
  - sys/bc_jupyter
  - sys/bc_matlab
  - sys/bc_codeserver
  - sys/bc_paraview
  - sys/bc_rstudio
  - sys/bc_tensorboard

File /etc/ood/config/apps/bc_desktop/ood_desktop.yml

---
title: "My Desktop"
cluster: "my_cluster"
attributes:
  desktop: "xfce"

File /etc/ood/config/apps/dashboard/initializers/ood.rb

Rails.application.config.after_initialize do
  OodFilesApp.candidate_favorite_paths.tap do |paths|
    paths << FavoritePath.new("/project/users/#{User.new.name}")
  end
end

File /etc/ood/config/nginx_stage.yml

---
pun_custom_env:
   OOD_DASHBOARD_TITLE: "My Open OnDemand"
   OOD_BRAND_BG_COLOR: "#004694"
   OOD_BRAND_LINK_ACTIVE_BG_COLOR: "#fff"

File /var/www/ood/apps/sys/bc_desktop/form.yml

---
attributes:
  desktop:
    label: "Desktop"
    widget: select
    options:
      - "xfce"
  bc_num_hours:
    value: 1
    min: 1
    max: 24
    step: 1
  bc_vnc_idle: 0
  bc_vnc_resolution:
    required: true
  node_type: null
  email:
    label: "Email"
form:
  - bc_vnc_idle
  - desktop
  - bc_num_hours
  - node_type
  - bc_vnc_resolution
  - email
  - bc_email_on_started

File /var/www/ood/apps/sys/bc_desktop/submit.yml.erb

---
batch_connect:
  template: vnc
script:
  email: <%= email %>
  native:
    <%- if bc_email_on_started == "1" -%>
    - "--mail-type=ALL"
    <%- end -%>
    - "--partition=normal"