SDF User Contributed Tutorials

User Tools

Site Tools

Trace: • the_metaarray
the_metaarray

Table of Contents

The MetaArray

The MetaArray is a group of multicore, multiraid, high performance nodes which provide computational and data storage/hosting services for MetaARPA members. The current disk quota for each user is 800 GB and will likely increase with time.

Acceptable Use Policy

Users of the MetaArray must agree to abide by the MetaArray Acceptable Use Policy

Getting Started

To enable your access to ma.sdf.org, run 'metaarray' at the shell. This will first generate a random login password followed by a random database password sent via local e-mail if you have a DBA membership. To connect, use 'ssh ma.sdf.org' (port 22 or 8080). Once you are connected, you can change your password with the 'passwd' command. If you ever forget your password, just run 'metaarray' again from any of the regular nodes of the SDF cluster.

Disk Layout

The MetaArray Logical Volume spans multiple Volume Groups across multiple Physical Volumes which are organized as multiple disk drives organized in multiple hardware RAID level 5 groups. To the user, this all appears to be under one contiguous file system: /meta

The user directory structure is as follows:

*/meta/initial/user(home directory, considered private)
*/meta/www/initial/user(web directory, considered public served via http)
*/meta/log(person website access log)
*/meta/mail/user(user's mailbox - see the Maildir tutorial)

Executables and Packages

Packages on the MetaArray are managed with pkgsrc and dpkg/apt-get. Nearly any program or package can be installed by making a request on the regular nodes of the SDF cluster via the 'bboard' REQUESTS.

The package directory structure is as follows:

* /bin, /usr/bin (apt-get repo destination directories)
* /usr/pkg/bin (pkgsrc destination directories)
* /usr/local/bin (MetaArray specific utilities)
* /meta/sdf/bin (compiled binaries installed by MetaARPA members)

Services and features

The MetaArray includes all features of the MetaARPA membership. However, some SDF specific utilities may not be available on the MetaArray because it is designed to be isolated from the SDF cluster. Some MetaArray specific services must be manually configured by the user.

*screen, tmuxTerminal managers
*mkcronUse 'crontab -e' to edit your crontab
*mkhomepgUse 'mkhomepg -t' on the SDF Cluster to toggle website hosting 'mkhomepg' on the MetaArray will set secure perms for /meta/www
*setvmailPut '$LOGNAME@ma.sdf.org' in your .forward on the SDF Cluster. Your default email address is $LOGNAME@SDF.ORG
*webmailBe sure to forward your mail, login via https://ma.sdf.org
*IMAPimap(143) and imaps(993) address is ma.sdf.org
*X11Connect with either ssh -X or ssh -Y to ma.sdf.org
*Port forwarding(via SSH)
*startsqlMySQL database (with DBA membership)

Personal Website Hosting

To host your personal website on the MetaArray, please do the following:

  • run 'mkhomepg' on the MetaArray. This creates the $HOME/html symlink and sets initial permissions.
  • upload your site to $HOME/html on the MetaArray.
  • run 'mkhomepg' again on the MetaArray. This sets proper permissions on your content.
  • run 'mkhomepg -t' on the SDF cluster to toggle your site to the MetaArray

It may take up to 10 minutes for DNS to update locally; however your ISP DNS server may cache this data for up to 12 hours so you won't see the change immediately. Use the “host” command on the cluster to see if DNS has changed, and “lynx” on the cluster to verify that the site loads: then be patient as DNS changes propagate.

* The DOCROOT for your directory defaults to no-read access, so you must have an index.html. Directory listings will work on subdirs within the DOCROOT. You can defeat this by running 'chmod g+r $HOME/html'

* https through a self signed certificate is only available to personal URLs under *.sdf.org

* Migrating your CGI scripts from the cluster to the MetaArray? You'll need to pay attention to the filename (only '.cgi' seems to work) and the path to the interpreter. For example, the following script will generate an actual webpage when served from the cluster, but when served from the MetaArray will prompt your browser for a location to save the script itself (first because the path to the interpreter is wrong, and second because the file extension is not recognized by nginx as needing the fastcgi module for its processing). 

#!/usr/pkg/bin/python3.9
# saved as $HOME/html/my_powerful.py
import cgi

print('Content-Type:text/html')
print()
print('<!DOCTYPE html>')
print('<html lang="en">')
print('<head>')
print('<title>My powerful script</title>')
print('</head>')
print('<body>')
print('Hello, World!')
print('</body>')
print('</html>')

To get this script to behave properly on the MetaArray, change the first line to #!/usr/bin/env python3.9, and rename the file as my_powerful.cgi .

Simple Personal Server Hosting

META users can run simple servers using the port that matches their user id. You may run a simple server process on the metaarray as long as it does not circumvent a membership and is a small server “of reasonable or less foot-print”. Find more detail and instructions in Meta Port Allocation

SDF Nextcloud

SDF makes available a Nextcloud 1) instance at https://ma.sdf.org/nc/.

To initialize your login, go to https://ma.sdf.org/nc/, click “Forgot password?”, and enter your SDF username.

In order to connect to your instance from a client, the “server” should be set to “https://ma.sdf.org/nc/”. (The client might complain about the *.sdf.org SSL certificate, but that's fine)

Using SDF Nextcloud on ShareX

If you'd like to use SDF's Nextcloud as a destination for ShareX then follow the steps below: note: This tutorial was written for 10.1

  • Configure ShareX Destination for SDF Nextcloud
  • Open the ShareX application window
  • Click on Destinations
  • Click on Destination Settings
  • Select on the left pane, Nextcloud (under File uploaders)
  • Enter:
  • Place a check next to:
    • Create shareable URL (Optional)
    • Ignore invalid SSL certificate (Required)
  • Close the settings window
  • Select from the menu: Image Uploader - File Uploader - Nextcloud

$Id: metaarray.html,v 1.38 2018/08/15 02:09:33 echosa Exp $ The MetaArray - traditional link (using RCS)

1)
Nextcloud -- Share and collaborate on documents, send and receive email, manage your calendar and have video chats without data leaks
the_metaarray.txt · Last modified: 2023/01/26 00:14 by peteyboy

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki