Bill Wang billhhh

I've used this in the past to grab offline copies of some of the Google API docs.

$ wget --no-parent --recursive --page-requisites --html-extension --convert-links "http://code.google.com/apis/gears/"

# --no-parent - Don't fetch anything in the parent directory of the URL specified
# --page-requisites - Download everything required for the page to display correctly
# --recursive - Recursively download (use default depth of 5)
# --html-extension - Add .html to downloaded html files (this isn't necessarily required for the google code example but might be useful on other sites)

--convert-links - Update links in the downloaded document to point to, either the downloaded copy of a file or a full URL

I recently happened upon an implementation of popen() (different API, same idea) using clone(2), and so I opened an issue requesting use of vfork(2) or posix_spawn() for portability. It turns out that on Linux there's an important advantage to using clone(2). I think I should capture the things I wrote there in a better place. A gist, a blog, whatever.

So here goes.

Long ago, I, like many Unix fans, thought that fork(2) and the fork-exec process spawning model were the greatest thing, and the Windows sucked for only having [exec*()](http://pubs.opengroup.org/onlinepubs/9699919

	# train_grpo.py
	#
	# See https://github.com/willccbb/verifiers for ongoing developments
	#
	import re
	import torch
	from datasets import load_dataset, Dataset
	from transformers import AutoTokenizer, AutoModelForCausalLM
	from peft import LoraConfig
	from trl import GRPOConfig, GRPOTrainer

	deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse
	deb-src http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse

	deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse
	deb-src http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse

	deb http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
	deb-src http://archive.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse

	deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse

	#sudo wget https://github.com/shiftkey/desktop/releases/download/release-2.1.0-linux1/GitHubDesktop-linux-2.1.0-linux1.deb
	#sudo gdebi GitHubDesktop-linux-2.1.0-linux1.deb

	# UPDATE (2021-03-05): Thanks to PaoloRanzi81's comment, the updated code is as follows https://gist.github.com/PaoloRanzi81

	sudo wget https://github.com/shiftkey/desktop/releases/download/release-2.6.3-linux1/GitHubDesktop-linux-2.6.3-linux1.deb
	### Uncomment below line if you have not installed gdebi-core before
	# sudo apt-get install gdebi-core
	sudo gdebi GitHubDesktop-linux-2.6.3-linux1.deb

	import nibabel as nib
	import numpy as np

	data = np.ones((32, 32, 15, 100), dtype=np.int16) # dummy data in numpy matrix
	img = nib.Nifti1Image(data, np.eye(4)) # Save axis for data (just identity)

	img.header.get_xyzt_units()
	img.to_filename(os.path.join('build','test4d.nii.gz')) # Save as NiBabel file

	import numpy as np
	from scipy.sparse import csc_matrix

	def pageRank(G, s = .85, maxerr = .001):
	"""
	Computes the pagerank for each of the n states.

	Used in webpage ranking and text summarization using unweighted
	or weighted transitions respectively.

	def contextual_loss(x, y, h=0.5):
	"""Computes contextual loss between x and y.

	Args:
	x: features of shape (N, C, H, W).
	y: features of shape (N, C, H, W).

	Returns:
	cx_loss = contextual loss between x and y (Eq (1) in the paper)
	"""

	#import random, math

	outputdebug = False

	def debug(msg):
	if outputdebug:
	print msg

	class Node():
	def __init__(self, key):

	# Imports
	from sklearn.datasets import fetch_olivetti_faces
	import numpy as np

	# Download Olivetti faces dataset
	olivetti = fetch_olivetti_faces()
	x = olivetti.images
	y = olivetti.target

	# Print info on shapes and reshape where necessary